{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/03_Earnings_Call_Analysis.ipynb)\n",
    "\n",
    "# Earnings Call Analysis with Docling, Semantica & AWS Neptune  \n",
    "MDA Space Ltd. — Q3 2025\n",
    "\n",
    "## Data Sources\n",
    "\n",
    "This notebook analyzes two financial documents from **MDA Space Ltd.** for Q3 2025:\n",
    "\n",
    "1. **Press Release** — Summary of financial results and management commentary  \n",
    "2. **Earnings Call Transcript** — Management presentation and analyst Q&A  \n",
    "\n",
    "Together, these documents provide both quantitative results and qualitative context.\n",
    "\n",
    "---\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates an end-to-end semantic pipeline for transforming\n",
    "unstructured financial documents into structured, queryable knowledge.\n",
    "\n",
    "**Docling** is used for high-fidelity document parsing. **Semantica** performs\n",
    "semantic extraction, cleaning, validation, and knowledge graph construction.\n",
    "The final knowledge graph is stored in **AWS Neptune** and used for hybrid\n",
    "retrieval, agent memory, and grounded question answering.\n",
    "\n",
    "---\n",
    "\n",
    "## End-to-End Workflow\n",
    "\n",
    "**Workflow:**  \n",
    "Dual PDF Input → Docling Parsing → Normalization & Chunking → Entity, Relation Extraction → Conflict Resolution & Deduplication → Knowledge Graph Construction → Amazon Neptune → GraphRAG → Agent Memory & Context → Strategic Q&A\n",
    "\n",
    "---\n",
    "\n",
    "## Pipeline Capabilities\n",
    "\n",
    "- High-fidelity PDF parsing (text, tables, structure)  \n",
    "- Semantic extraction of entities, and relationships\n",
    "- Conflict detection and resolution with confidence awareness  \n",
    "- Entity deduplication and canonicalization  \n",
    "- Knowledge graph construction and validation  \n",
    "- Persistent graph storage in **AWS Neptune** (IAM, OpenCypher)  \n",
    "- Hybrid retrieval using **GraphRAG** (vector + graph)  \n",
    "- Long-term agent memory and unified context management  \n",
    "- Grounded LLM-based question answering  \n",
    "- Structured export to JSON and RDF formats  \n",
    "\n",
    "---\n",
    "\n",
    "## Outcome\n",
    "\n",
    "The output is a cleaned, deduplicated knowledge graph stored in **AWS Neptune**,\n",
    "along with supporting context for hybrid retrieval and question answering.\n",
    "This enables reliable financial analysis and downstream applications built\n",
    "on structured, traceable knowledge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n"
     ]
    }
   ],
   "source": [
    "!pip install -qU semantica docling pdfplumber groq\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✓ Groq LLM initialized: llama-3.1-8b-instant\n"
     ]
    }
   ],
   "source": [
    "from getpass import getpass\n",
    "from semantica.llms import Groq\n",
    "\n",
    "GROQ_API_KEY = getpass(\"Enter your GROQ API key: \")\n",
    "\n",
    "if not GROQ_API_KEY:\n",
    "    raise ValueError(\"GROQ API key is required\")\n",
    "\n",
    "groq_llm = Groq(\n",
    "    model=\"llama-3.1-8b-instant\",\n",
    "    api_key=GROQ_API_KEY,\n",
    ")\n",
    "\n",
    "print(f\"✓ Groq LLM initialized: {groq_llm.model}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Parse PDF with Docling\n",
    "\n",
    "Parse earnings call PDF and extract financial tables using DoclingParser.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "DoclingParser requires the 'docling' package to be installed and working.\n\nError: [WinError 1114] A dynamic link library (DLL) initialization routine failed. Error loading \"c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\torch\\lib\\c10.dll\" or one of its dependencies.\n\nInstall it with: pip install docling",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[3], line 5\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01mpathlib\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m Path\n\u001b[0;32m      3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01msemantica\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mparse\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m DoclingParser\n\u001b[1;32m----> 5\u001b[0m parser \u001b[38;5;241m=\u001b[39m \u001b[43mDoclingParser\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m      7\u001b[0m PRESS_RELEASE_URL \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttps://filecache.investorroom.com/mr5ircnw_mda/677/MDA_Space_Ltd_Q3_2025_Press_Release_Nov_14_2025_FINAL.pdf\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m      8\u001b[0m TRANSCRIPT_URL \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttps://filecache.investorroom.com/mr5ircnw_mda/681/MDA\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m20Space\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m20Ltd.\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m20Q3\u001b[39m\u001b[38;5;132;01m%202025%\u001b[39;00m\u001b[38;5;124m20Earnings\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m20Conference\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m20Call\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m20Transcript\u001b[39m\u001b[38;5;132;01m%20%\u001b[39;00m\u001b[38;5;124m28November\u001b[39m\u001b[38;5;132;01m%2014%\u001b[39;00m\u001b[38;5;124m202025\u001b[39m\u001b[38;5;124m%\u001b[39m\u001b[38;5;124m29.pdf\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
      "File \u001b[1;32mc:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\semantica\\parse\\__init__.py:198\u001b[0m, in \u001b[0;36mDoclingParser.__init__\u001b[1;34m(self, **config)\u001b[0m\n\u001b[0;32m    196\u001b[0m     error_msg \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124mError: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mimport_error_msg\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    197\u001b[0m error_msg \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124mInstall it with: pip install docling\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m--> 198\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m(error_msg)\n",
      "\u001b[1;31mImportError\u001b[0m: DoclingParser requires the 'docling' package to be installed and working.\n\nError: [WinError 1114] A dynamic link library (DLL) initialization routine failed. Error loading \"c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\torch\\lib\\c10.dll\" or one of its dependencies.\n\nInstall it with: pip install docling"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from pathlib import Path\n",
    "from semantica.parse import DoclingParser\n",
    "\n",
    "parser = DoclingParser()\n",
    "\n",
    "PRESS_RELEASE_URL = \"https://filecache.investorroom.com/mr5ircnw_mda/677/MDA_Space_Ltd_Q3_2025_Press_Release_Nov_14_2025_FINAL.pdf\"\n",
    "TRANSCRIPT_URL = \"https://filecache.investorroom.com/mr5ircnw_mda/681/MDA%20Space%20Ltd.%20Q3%202025%20Earnings%20Conference%20Call%20Transcript%20%28November%2014%202025%29.pdf\"\n",
    "\n",
    "download_dir = Path(\"downloads\")\n",
    "download_dir.mkdir(exist_ok=True)\n",
    "\n",
    "press_release_pdf = download_dir / \"mda_space_q3_2025_press_release.pdf\"\n",
    "transcript_pdf = download_dir / \"mda_space_q3_2025_transcript.pdf\"\n",
    "\n",
    "if not press_release_pdf.exists():\n",
    "    press_release_pdf.write_bytes(requests.get(PRESS_RELEASE_URL).content)\n",
    "\n",
    "if not transcript_pdf.exists():\n",
    "    transcript_pdf.write_bytes(requests.get(TRANSCRIPT_URL).content)\n",
    "\n",
    "try:\n",
    "    press_release = parser.parse(press_release_pdf)\n",
    "    transcript = parser.parse(transcript_pdf)\n",
    "except Exception as e:\n",
    "    print(\"Parsing failed\")\n",
    "    print(e)\n",
    "    print(\"Using fallback empty documents.\")\n",
    "    press_release = {\"full_text\": \"\", \"tables\": []}\n",
    "    transcript = {\"full_text\": \"\", \"tables\": []}\n",
    "\n",
    "parsed_doc = {\n",
    "    \"full_text\": (\n",
    "        \"# Press Release\\n\\n\"\n",
    "        f\"{press_release['full_text']}\\n\\n\"\n",
    "        \"# Transcript\\n\\n\"\n",
    "        f\"{transcript['full_text']}\"\n",
    "    ),\n",
    "    \"tables\": press_release[\"tables\"] + transcript[\"tables\"],\n",
    "    \"metadata\": {\n",
    "        \"title\": \"MDA Space Ltd. Q3 2025 Earnings Analysis\",\n",
    "        \"company\": \"MDA Space Ltd.\",\n",
    "        \"quarter\": \"Q3 2025\",\n",
    "        \"date\": \"November 14, 2025\",\n",
    "    },\n",
    "}\n",
    "\n",
    "print(\"Parsing completed\")\n",
    "print(\"Documents processed: 2\")\n",
    "print(\"Tables extracted:\", len(parsed_doc[\"tables\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Normalize Text\n",
    "\n",
    "Normalize extracted text using TextNormalizer for consistent processing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "\n",
    "normalized_text = normalizer.normalize(\n",
    "    parsed_doc[\"full_text\"],\n",
    "    clean_html=False,\n",
    "    remove_extra_whitespace=False,\n",
    "    lowercase=False,\n",
    ")\n",
    "\n",
    "print(\"Normalization completed\")\n",
    "print(\"Normalized text length:\", len(normalized_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Split Text into Chunks\n",
    "\n",
    "Split the normalized text into overlapping chunks to enable scalable and accurate entity and relation extraction.\n",
    "This step prepares the text for LLM-based semantic processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.split import TextSplitter\n",
    "\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 250\n",
    "\n",
    "splitter = TextSplitter(\n",
    "    method=\"recursive\",\n",
    "    chunk_size=CHUNK_SIZE,\n",
    "    chunk_overlap=CHUNK_OVERLAP,\n",
    ")\n",
    "\n",
    "chunks = splitter.split(normalized_text)\n",
    "\n",
    "def get_chunk_text(chunk):\n",
    "    return getattr(chunk, \"content\", getattr(chunk, \"text\", \"\"))\n",
    "\n",
    "print(\"Chunking completed\")\n",
    "print(\"Total chunks:\", len(chunks))\n",
    "print(\"Sample chunk:\")\n",
    "print(get_chunk_text(chunks[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Extract Entities\n",
    "\n",
    "Extract entities (organizations, people, financial terms) using NERExtractor with Groq LLM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "ner = NERExtractor(\n",
    "    method=\"llm\",\n",
    "    provider=\"groq\",\n",
    "    llm_model=\"llama-3.1-8b-instant\",\n",
    "    temperature=0.0,\n",
    "    api_key=GROQ_API_KEY,\n",
    "    max_retries=3,\n",
    ")\n",
    "\n",
    "ENTITY_TYPES = [\"ORGANIZATION\", \"PERSON\", \"MONEY\", \"PERCENT\", \"DATE\", \"EVENT\"]\n",
    "\n",
    "all_entities = [\n",
    "    e\n",
    "    for c in chunks\n",
    "    for e in ner.extract_entities(\n",
    "        get_chunk_text(c),\n",
    "        entity_types=ENTITY_TYPES,\n",
    "    )\n",
    "    if get_chunk_text(c).strip()\n",
    "]\n",
    "\n",
    "print(\"Entities:\", len(all_entities))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Extract Financial Metrics\n",
    "\n",
    "Extract financial metrics (money, percentages, dates) from text and tables.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FINANCIAL_ENTITY_TYPES = [\n",
    "    \"MONEY\", \"CURRENCY\", \"PERCENT\", \"PERCENTAGE\",\n",
    "    \"QUANTITY\", \"CARDINAL\",\n",
    "]\n",
    "\n",
    "financial_entities = []\n",
    "\n",
    "for chunk in chunks:\n",
    "    text = get_chunk_text(chunk)\n",
    "    if text.strip():\n",
    "        financial_entities += ner.extract_entities(\n",
    "            text,\n",
    "            entity_types=FINANCIAL_ENTITY_TYPES,\n",
    "        )\n",
    "\n",
    "money, percentages, quantities = [], [], []\n",
    "\n",
    "for e in financial_entities:\n",
    "    label = e.label.upper()\n",
    "    if label in (\"MONEY\", \"CURRENCY\"):\n",
    "        money.append(e.text)\n",
    "    elif label in (\"PERCENT\", \"PERCENTAGE\"):\n",
    "        percentages.append(e.text)\n",
    "    elif label in (\"CARDINAL\", \"QUANTITY\"):\n",
    "        quantities.append(e.text)\n",
    "\n",
    "print(\"\\nFinancial entity extraction completed\")\n",
    "print(\"Total financial entities:\", len(financial_entities))\n",
    "print(\"Money:\", len(money))\n",
    "print(\"Percentages:\", len(percentages))\n",
    "print(\"Quantities:\", len(quantities))\n",
    "\n",
    "if financial_entities:\n",
    "    print(\"Sample:\", f\"{financial_entities[0].text} ({financial_entities[0].label})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Extract Relationships\n",
    "\n",
    "Extract relationships between entities using RelationExtractor with Groq LLM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from concurrent.futures import ThreadPoolExecutor, TimeoutError\n",
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "MAX_ENTITIES = 30\n",
    "CHUNK_TIMEOUT = 60\n",
    "\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"llm\",\n",
    "    confidence_threshold=0.6,\n",
    "    relation_types=[\n",
    "        \"HAS_REVENUE\",\n",
    "        \"HAS_GROWTH\",\n",
    "        \"REPORTS\",\n",
    "        \"PROVIDES_GUIDANCE\",\n",
    "        \"IN_QUARTER\",\n",
    "        \"FOR_PERIOD\",\n",
    "        \"RELATED_TO\",\n",
    "    ],\n",
    "    provider=\"groq\",\n",
    "    llm_model=\"llama-3.1-8b-instant\",\n",
    "    api_key=GROQ_API_KEY,\n",
    "    temperature=0.0,\n",
    "    verbose=False,\n",
    ")\n",
    "\n",
    "\n",
    "def filter_entities(text, entities):\n",
    "    t = text.lower()\n",
    "    return [e for e in entities if e.text.lower() in t]\n",
    "\n",
    "\n",
    "def process_chunk(idx, chunk, total):\n",
    "    text = get_chunk_text(chunk).strip()\n",
    "\n",
    "    remaining = total - (idx + 1)\n",
    "\n",
    "    if not text:\n",
    "        print(f\"Chunk {idx+1}/{total} | remaining {remaining} | skipped (empty)\")\n",
    "        return []\n",
    "\n",
    "    chunk_entities = filter_entities(text, all_entities)[:MAX_ENTITIES]\n",
    "\n",
    "    if len(chunk_entities) < 2:\n",
    "        print(\n",
    "            f\"Chunk {idx+1}/{total} | remaining {remaining} | \"\n",
    "            f\"skipped (entities={len(chunk_entities)})\"\n",
    "        )\n",
    "        return []\n",
    "\n",
    "    print(\n",
    "        f\"Chunk {idx+1}/{total} | remaining {remaining} | \"\n",
    "        f\"entities={len(chunk_entities)}\"\n",
    "    )\n",
    "\n",
    "    return relation_extractor.extract_relations(\n",
    "        text=text,\n",
    "        entities=chunk_entities,\n",
    "        verbose=False,\n",
    "    )\n",
    "\n",
    "\n",
    "relationships = []\n",
    "total_chunks = len(chunks)\n",
    "\n",
    "with ThreadPoolExecutor(max_workers=1) as executor:\n",
    "    for i, c in enumerate(chunks):\n",
    "        future = executor.submit(process_chunk, i, c, total_chunks)\n",
    "\n",
    "        try:\n",
    "            rels = future.result(timeout=CHUNK_TIMEOUT)\n",
    "            relationships.extend(rels)\n",
    "            print(f\"  relations={len(rels)}\")\n",
    "\n",
    "        except TimeoutError:\n",
    "            remaining = total_chunks - (i + 1)\n",
    "            print(\n",
    "                f\"Chunk {i+1}/{total_chunks} | remaining {remaining} | timed out\"\n",
    "            )\n",
    "\n",
    "        except Exception as e:\n",
    "            remaining = total_chunks - (i + 1)\n",
    "            print(\n",
    "                f\"Chunk {i+1}/{total_chunks} | remaining {remaining} | failed: {e}\"\n",
    "            )\n",
    "\n",
    "print(f\"Done {total_chunks}/{total_chunks}\")\n",
    "print(f\"Total relationships: {len(relationships)}\")\n",
    "\n",
    "if relationships:\n",
    "    for r in relationships[:10]:\n",
    "        print(f\"{r.subject.text} → {r.predicate} → {r.object.text}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Detect Conflicts\n",
    "\n",
    "Detect conflicts in extracted entities and relationships using ConflictDetector.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import SourceTracker, SourceReference, ConflictDetector\n",
    "\n",
    "source_tracker = SourceTracker()\n",
    "\n",
    "conflict_detector = ConflictDetector(\n",
    "    source_tracker=source_tracker,\n",
    "    similarity_threshold=0.8,\n",
    "    confidence_threshold=0.7,\n",
    ")\n",
    "\n",
    "entities = all_entities\n",
    "extracted_relationships = relationships\n",
    "\n",
    "for e in entities:\n",
    "    entity_id = getattr(e, \"id\", None) or e.text\n",
    "    source_tracker.track_property_source(\n",
    "        entity_id=entity_id,\n",
    "        property_name=\"name\",\n",
    "        value=e.text,\n",
    "        source=SourceReference(\n",
    "            document=\"earnings_call\",\n",
    "            timestamp=\"2024-Q1\",\n",
    "            metadata={\"entity_type\": getattr(e, \"label\", \"UNKNOWN\")},\n",
    "        ),\n",
    "    )\n",
    "\n",
    "entity_records = [\n",
    "    {\n",
    "        \"id\": getattr(e, \"id\", None) or e.text,\n",
    "        \"name\": e.text,\n",
    "    }\n",
    "    for e in entities\n",
    "]\n",
    "\n",
    "entity_value_conflicts = conflict_detector.detect_value_conflicts(\n",
    "    entity_records,\n",
    "    property_name=\"name\",\n",
    ")\n",
    "\n",
    "normalized_relationships = [\n",
    "    {\n",
    "        \"id\": getattr(r, \"id\", None),\n",
    "        \"source_id\": getattr(r.subject, \"id\", None) or r.subject.text,\n",
    "        \"target_id\": getattr(r.object, \"id\", None) or r.object.text,\n",
    "        \"type\": r.predicate,\n",
    "        \"confidence\": getattr(r, \"confidence\", 1.0),\n",
    "        \"metadata\": {},\n",
    "    }\n",
    "    for r in extracted_relationships\n",
    "]\n",
    "\n",
    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(\n",
    "    normalized_relationships\n",
    ")\n",
    "\n",
    "print(\"Conflict detection completed\")\n",
    "print(\"Entity value conflicts:\", len(entity_value_conflicts))\n",
    "print(\"Relationship conflicts:\", len(relationship_conflicts))\n",
    "\n",
    "if entity_value_conflicts:\n",
    "    print(\"\\nSample entity conflict:\")\n",
    "    print(entity_value_conflicts[0])\n",
    "\n",
    "if relationship_conflicts:\n",
    "    print(\"\\nSample relationship conflict:\")\n",
    "    print(relationship_conflicts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Resolve Conflicts\n",
    "\n",
    "Resolve detected conflicts using ConflictResolver with voting strategy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictResolver\n",
    "\n",
    "conflict_resolver = ConflictResolver(\n",
    "    default_strategy=\"voting\",\n",
    "    source_tracker=source_tracker,\n",
    ")\n",
    "\n",
    "resolved_entity_value_conflicts = []\n",
    "resolved_relationship_conflicts = []\n",
    "\n",
    "for conflict in entity_value_conflicts:\n",
    "    resolved_entity_value_conflicts.append(\n",
    "        conflict_resolver.resolve_conflict(\n",
    "            conflict,\n",
    "            strategy=\"voting\",\n",
    "        )\n",
    "    )\n",
    "\n",
    "for conflict in relationship_conflicts:\n",
    "    resolved_relationship_conflicts.append(\n",
    "        conflict_resolver.resolve_conflict(\n",
    "            conflict,\n",
    "            strategy=\"voting\",\n",
    "        )\n",
    "    )\n",
    "\n",
    "print(\"Conflict resolution completed\")\n",
    "print(\"Entity value conflicts resolved:\", len(resolved_entity_value_conflicts))\n",
    "print(\"Relationship conflicts resolved:\", len(resolved_relationship_conflicts))\n",
    "\n",
    "if resolved_entity_value_conflicts:\n",
    "    print(\"\\nSample resolved entity conflict:\")\n",
    "    print(resolved_entity_value_conflicts[0])\n",
    "\n",
    "if resolved_relationship_conflicts:\n",
    "    print(\"\\nSample resolved relationship conflict:\")\n",
    "    print(resolved_relationship_conflicts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Deduplicate Entities\n",
    "\n",
    "Detect and merge duplicate entities using DuplicateDetector and EntityMerger.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.deduplication import DuplicateDetector, EntityMerger\n",
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "raw = []\n",
    "for i, e in enumerate(entities):\n",
    "    raw.append({\n",
    "        \"id\": getattr(e, \"id\", None) or f\"entity_{i}_{getattr(e, 'text', str(e))}\",\n",
    "        \"name\": (getattr(e, \"text\", getattr(e, \"name\", \"\")) or \"\").strip(),\n",
    "        \"type\": getattr(e, \"label\", \"UNKNOWN\"),\n",
    "        \"confidence\": float(getattr(e, \"confidence\", 1.0) or 1.0),\n",
    "        \"metadata\": getattr(e, \"metadata\", {}),\n",
    "    })\n",
    "\n",
    "filtered = [r for r in raw if r[\"name\"] and len(r[\"name\"]) >= 3]\n",
    "\n",
    "collapsed = {}\n",
    "for ent in filtered:\n",
    "    key = (ent[\"type\"], ent[\"name\"].lower())\n",
    "    best = collapsed.get(key)\n",
    "    if best is None or ent[\"confidence\"] > best[\"confidence\"]:\n",
    "        collapsed[key] = ent\n",
    "\n",
    "entity_dicts = list(collapsed.values())\n",
    "\n",
    "detector = DuplicateDetector(\n",
    "    similarity_threshold=0.96,\n",
    "    confidence_threshold=0.92,\n",
    "    use_clustering=True,\n",
    ")\n",
    "\n",
    "detector.detect_duplicate_groups(entity_dicts)\n",
    "\n",
    "merger = EntityMerger(\n",
    "    preserve_provenance=True,\n",
    "    detector={\n",
    "        \"similarity_threshold\": 0.96,\n",
    "        \"confidence_threshold\": 0.92,\n",
    "        \"use_clustering\": True,\n",
    "    },\n",
    "    strategy={\"default_strategy\": \"keep_most_complete\"},\n",
    ")\n",
    "\n",
    "merge_operations = merger.merge_duplicates(\n",
    "    entities=entity_dicts,\n",
    "    strategy=\"keep_most_complete\",\n",
    ")\n",
    "\n",
    "deduplicated_entities = [op.merged_entity for op in merge_operations] or entity_dicts\n",
    "\n",
    "entity_id_mapping = {}\n",
    "for op in merge_operations:\n",
    "    mid = op.merged_entity[\"id\"]\n",
    "    for sid in op.source_ids:\n",
    "        entity_id_mapping[sid] = mid\n",
    "\n",
    "deduplicated_relationships = []\n",
    "for rel in normalized_relationships:\n",
    "    s = entity_id_mapping.get(rel[\"source_id\"], rel[\"source_id\"])\n",
    "    t = entity_id_mapping.get(rel[\"target_id\"], rel[\"target_id\"])\n",
    "    if s != t:\n",
    "        r = rel.copy()\n",
    "        r[\"source_id\"], r[\"target_id\"] = s, t\n",
    "        deduplicated_relationships.append(r)\n",
    "\n",
    "print({\n",
    "    \"time_seconds\": round(time.time() - start_time, 2),\n",
    "    \"entities_in\": len(raw),\n",
    "    \"entities_after_filter\": len(filtered),\n",
    "    \"entities_after_exact\": len(entity_dicts),\n",
    "    \"entities_out\": len(deduplicated_entities),\n",
    "    \"duplicates_removed\": len(entity_dicts) - len(deduplicated_entities),\n",
    "    \"relationships_updated\": len(deduplicated_relationships),\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Build Knowledge Graph\n",
    "\n",
    "Build knowledge graph from cleaned entities, relationships, and triplets using GraphBuilder.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "# Deduplication is already done; avoid additional entity resolution/merging\n",
    "graph_builder = GraphBuilder(\n",
    "    merge_entities=False,\n",
    "    entity_resolution_strategy=\"none\",\n",
    ")\n",
    "\n",
    "final_relationships = deduplicated_relationships\n",
    "\n",
    "kg_data = {\n",
    "    \"entities\": deduplicated_entities,\n",
    "    \"relationships\": final_relationships,\n",
    "    \"metadata\": {\n",
    "        \"source\": \"earnings_call_transcript\",\n",
    "        \"extraction_method\": \"Groq LLM\",\n",
    "    },\n",
    "}\n",
    "\n",
    "knowledge_graph = graph_builder.build(\n",
    "    sources=[kg_data],\n",
    "    merge_entities=False,\n",
    ")\n",
    "\n",
    "print(\"Knowledge graph build completed (no additional merging)\")\n",
    "print(\"Final entities:\", len(knowledge_graph.get(\"entities\", [])))\n",
    "print(\"Final relationships:\", len(knowledge_graph.get(\"relationships\", [])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 10: Analyze Knowledge Graph\n",
    "\n",
    "This step evaluates the structure and quality of the knowledge graph.\n",
    "\n",
    "- **Centrality**  \n",
    "  Identifies the most influential entities based on connectivity.\n",
    "\n",
    "- **Communities**  \n",
    "  Groups related entities to reveal themes such as business units, markets, or topics.\n",
    "\n",
    "- **Connectivity**  \n",
    "  Shows how well the graph is linked and whether information is fragmented.\n",
    "\n",
    "These metrics help validate extraction quality and guide downstream analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphAnalyzer\n",
    "\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "\n",
    "analysis = graph_analyzer.analyze_graph(knowledge_graph)\n",
    "centrality = graph_analyzer.calculate_centrality(\n",
    "    knowledge_graph,\n",
    "    method=\"degree\",\n",
    ")\n",
    "communities = graph_analyzer.detect_communities(\n",
    "    knowledge_graph,\n",
    "    algorithm=\"louvain\",\n",
    ")\n",
    "connectivity = graph_analyzer.analyze_connectivity(knowledge_graph)\n",
    "metrics = graph_analyzer.compute_metrics(knowledge_graph)\n",
    "\n",
    "num_communities = len(communities.get(\"communities\", []))\n",
    "\n",
    "print(\"Graph analysis completed\")\n",
    "print(\"Communities:\", num_communities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 11: Persist Knowledge Graph in Amazon Neptune\n",
    "\n",
    "After cleaning, conflict resolution, and deduplication, the final step is to\n",
    "persist the **canonical knowledge graph** into a production graph database.\n",
    "\n",
    "Semantica integrates with **Amazon Neptune** to provide a secure, scalable,\n",
    "and query-efficient backend for long-lived knowledge graphs.\n",
    "\n",
    "- **Canonical Storage**  \n",
    "  Only deduplicated entities and resolved relationships are written to Neptune.\n",
    "\n",
    "- **Secure Access**  \n",
    "  Uses AWS IAM authentication (SigV4) for production-grade security.\n",
    "\n",
    "- **Flexible Graph Model**  \n",
    "  Supports property graphs (OpenCypher / Gremlin) and RDF (SPARQL).\n",
    "\n",
    "- **Efficient Querying**  \n",
    "  Leverages the Bolt protocol for low-latency graph queries and traversal.\n",
    "\n",
    "- **Production Ready**  \n",
    "  Designed for compliance, provenance, and downstream analytics.\n",
    "\n",
    "This step enables durable storage, rich querying, and integration with\n",
    "analytics and applications on top of the extracted knowledge graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"NEPTUNE_ENDPOINT\"] = \"your-cluster.us-east-1.neptune.amazonaws.com\"\n",
    "os.environ[\"NEPTUNE_PORT\"] = \"8182\"\n",
    "os.environ[\"AWS_REGION\"] = \"us-east-1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.graph_store import GraphStore\n",
    "import os\n",
    "\n",
    "neptune_store = GraphStore(\n",
    "    backend=\"neptune\",\n",
    "    endpoint=os.environ[\"NEPTUNE_ENDPOINT\"],\n",
    "    port=int(os.environ.get(\"NEPTUNE_PORT\", 8182)),\n",
    "    region=os.environ[\"AWS_REGION\"],\n",
    "    iam_auth=True,\n",
    ")\n",
    "\n",
    "neptune_store.connect()\n",
    "print(\"Connected to AWS Neptune\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for entity in knowledge_graph.get(\"entities\", []):\n",
    "    neptune_store.create_node(\n",
    "        labels=[entity.get(\"type\", \"Entity\")],\n",
    "        properties=entity,\n",
    "    )\n",
    "\n",
    "for rel in knowledge_graph.get(\"relationships\", []):\n",
    "    neptune_store.create_relationship(\n",
    "        start_node_id=rel[\"source\"],\n",
    "        end_node_id=rel[\"target\"],\n",
    "        rel_type=rel[\"predicate\"],\n",
    "        properties=rel.get(\"metadata\", {}),\n",
    "    )\n",
    "\n",
    "print(\"Knowledge graph populated to AWS Neptune\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify data in Neptune\n",
    "results = neptune_store.execute_query(\n",
    "    \"MATCH (n) RETURN count(n) AS node_count\"\n",
    ")\n",
    "print(\"Total nodes:\", results.get(\"records\", [{}])[0].get(\"node_count\"))\n",
    "\n",
    "results = neptune_store.execute_query(\n",
    "    \"MATCH ()-[r]->() RETURN count(r) AS rel_count\"\n",
    ")\n",
    "print(\"Total relationships:\", results.get(\"records\", [{}])[0].get(\"rel_count\"))\n",
    "\n",
    "# Sample query: list a few entities\n",
    "results = neptune_store.execute_query(\n",
    "    \"MATCH (n) RETURN labels(n), n.name LIMIT 5\"\n",
    ")\n",
    "\n",
    "print(\"Sample nodes:\")\n",
    "for r in results.get(\"records\", []):\n",
    "    print(r)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 12: Context Retrieval\n",
    "\n",
    "Set up hybrid retrieval (vector + graph) using ContextRetriever for GraphRAG queries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from semantica.vector_store import VectorStore\n",
    "from semantica.context import ContextRetriever\n",
    "\n",
    "if 'chunks' not in locals() or not chunks:\n",
    "    raise ValueError(\"Chunks not found. Please run Step 3 first.\")\n",
    "\n",
    "# Extract text content safely\n",
    "chunk_texts = [getattr(c, \"content\", getattr(c, \"text\", \"\")) for c in chunks]\n",
    "chunk_metadatas = [\n",
    "    {\n",
    "        \"source\": \"earnings_call\", \n",
    "        \"type\": \"transcript\", \n",
    "        \"chunk_index\": i,\n",
    "        **(getattr(c, \"metadata\", {}) or {})\n",
    "    }\n",
    "    for i, c in enumerate(chunks)\n",
    "]\n",
    "\n",
    "# Initialize Vector Store (Optimized for Speed)\n",
    "# dimension=384 matches the default fast model (BAAI/bge-small-en-v1.5)\n",
    "vector_store = VectorStore(\n",
    "    backend=\"faiss\", \n",
    "    dimension=384,  \n",
    "    max_workers=16\n",
    ")\n",
    "\n",
    "print(f\"Storing {len(chunks)} chunks with high-performance settings...\")\n",
    "start_time = time.time()\n",
    "\n",
    "# Store in large batches with parallel processing\n",
    "vector_ids = vector_store.add_documents(\n",
    "    documents=chunk_texts,\n",
    "    metadata=chunk_metadatas,\n",
    "    batch_size=128,\n",
    "    parallel=True\n",
    ")\n",
    "\n",
    "print(f\"✅ Stored in {time.time() - start_time:.2f}s\")\n",
    "\n",
    "# Initialize Hybrid Retriever\n",
    "context_retriever = ContextRetriever(\n",
    "    knowledge_graph=knowledge_graph, # Assumes knowledge_graph exists\n",
    "    vector_store=vector_store,\n",
    "    hybrid_alpha=0.6,\n",
    "    use_graph_expansion=True,\n",
    "    max_expansion_hops=2,\n",
    ")\n",
    "\n",
    "# Test Retrieval\n",
    "queries = [\n",
    "    \"What was the company's revenue guidance?\",\n",
    "    \"What were the key financial metrics discussed?\",\n",
    "]\n",
    "\n",
    "retrieved_contexts = []\n",
    "for query in queries:\n",
    "    results = context_retriever.retrieve(\n",
    "        query=query,\n",
    "        max_results=3,\n",
    "        min_relevance_score=0.2,\n",
    "    )\n",
    "    retrieved_contexts.append(results)\n",
    "\n",
    "print(\"Hybrid GraphRAG configured\")\n",
    "print(\"Sample results:\", len(retrieved_contexts[0]) if retrieved_contexts else 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 13: Agent Memory (Long-Term Context)\n",
    "\n",
    "This step enables long-term memory for agents by storing important facts,\n",
    "metrics, and entities extracted from the knowledge graph.\n",
    "\n",
    "- **Semantic Memory Storage**  \n",
    "  Stores structured memories enriched with entities and relationships,\n",
    "  not just raw text.\n",
    "\n",
    "- **Hybrid Recall**  \n",
    "  Combines vector similarity with graph structure for accurate retrieval.\n",
    "\n",
    "- **Time-Bound Retention**  \n",
    "  Supports memory expiration policies for freshness and governance.\n",
    "\n",
    "- **Agent-Ready Context**  \n",
    "  Allows agents to recall prior earnings, metrics, and entities across sessions.\n",
    "\n",
    "Agent Memory turns one-off analysis into **persistent, reusable intelligence**\n",
    "for downstream agents and workflows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.context import AgentMemory\n",
    "\n",
    "agent_memory = AgentMemory(\n",
    "    vector_store=vector_store,\n",
    "    knowledge_graph=knowledge_graph,\n",
    "    retention_days=30,\n",
    ")\n",
    "\n",
    "entity_count = len(knowledge_graph.get(\"entities\", []))\n",
    "relationship_count = len(knowledge_graph.get(\"relationships\", []))\n",
    "\n",
    "memory_contents = [\n",
    "    f\"Earnings call transcript: {parsed_doc['metadata'].get('title', 'Earnings Call')}\",\n",
    "    f\"Graph entities: {entity_count}\",\n",
    "    f\"Graph relationships: {relationship_count}\",\n",
    "]\n",
    "\n",
    "memory_ids = []\n",
    "\n",
    "for content in memory_contents:\n",
    "    memory_ids.append(\n",
    "        agent_memory.store(\n",
    "            content=content,\n",
    "            metadata={\"source\": \"earnings_call\", \"type\": \"analysis\"},\n",
    "            extract_entities=True,\n",
    "            extract_relationships=True,\n",
    "        )\n",
    "    )\n",
    "\n",
    "financial_memories = agent_memory.retrieve(\n",
    "    query=\"financial metrics and earnings\",\n",
    "    max_results=5,\n",
    ")\n",
    "\n",
    "memory_stats = agent_memory.get_statistics()\n",
    "\n",
    "print(\"Agent memory configured\")\n",
    "print(\"Memories stored:\", len(memory_ids))\n",
    "print(\"Total memories:\", memory_stats.get(\"total_memories\", 0))\n",
    "print(\"Retrieved memories:\", len(financial_memories))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 14: Agent Context\n",
    "\n",
    "**AgentContext** provides a unified context layer that combines **vector-based RAG**\n",
    "with **graph-based GraphRAG** for grounded and explainable retrieval.\n",
    "\n",
    "### Key Controls\n",
    "- **Graph Expansion**  \n",
    "  Uses connected entities and relationships from the knowledge graph to\n",
    "  expand context beyond direct text matches.\n",
    "\n",
    "- **Hybrid Alpha**  \n",
    "  Balances text similarity with graph structure. Lower values favor text;\n",
    "  higher values favor graph reasoning.\n",
    "\n",
    "- **Expansion Hops**  \n",
    "  Limits how far context can expand through the graph. Fewer hops keep\n",
    "  results focused; more hops increase coverage.\n",
    "\n",
    "AgentContext enables agents to reason over both **documents** and\n",
    "**knowledge graphs** through a single interface.\n",
    "\n",
    "\n",
    "### AgentContext Parameters\n",
    "\n",
    "- **vector_store** – Vector search over unstructured text  \n",
    "- **knowledge_graph** – Structured entities and relationships  \n",
    "- **use_graph_expansion** – Enable GraphRAG (graph-based context expansion)  \n",
    "- **max_expansion_hops** – How far to traverse the graph  \n",
    "- **hybrid_alpha** – Balance between vector and graph relevance  \n",
    "- **retention_days** – How long context is kept  \n",
    "\n",
    "**Store options**\n",
    "- **link_entities** – Link to existing graph nodes  \n",
    "\n",
    "**Retrieve options**\n",
    "- **max_results** – Number of results returned  \n",
    "- **expand_graph** – Expand context via the graph  \n",
    "- **include_entities** – Return related entities  \n",
    "\n",
    "AgentContext unifies **memory, GraphRAG, and retrieval** in one interface.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'vector_store' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[4], line 4\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01msemantica\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcontext\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m AgentContext\n\u001b[0;32m      3\u001b[0m agent_context \u001b[38;5;241m=\u001b[39m AgentContext(\n\u001b[1;32m----> 4\u001b[0m     vector_store\u001b[38;5;241m=\u001b[39m\u001b[43mvector_store\u001b[49m,\n\u001b[0;32m      5\u001b[0m     knowledge_graph\u001b[38;5;241m=\u001b[39mknowledge_graph,\n\u001b[0;32m      6\u001b[0m     use_graph_expansion\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[0;32m      7\u001b[0m     max_expansion_hops\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m2\u001b[39m,\n\u001b[0;32m      8\u001b[0m     hybrid_alpha\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0.6\u001b[39m,\n\u001b[0;32m      9\u001b[0m     retention_days\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m30\u001b[39m,\n\u001b[0;32m     10\u001b[0m )\n\u001b[0;32m     12\u001b[0m memory_id \u001b[38;5;241m=\u001b[39m agent_context\u001b[38;5;241m.\u001b[39mstore(\n\u001b[0;32m     13\u001b[0m     content\u001b[38;5;241m=\u001b[39mparsed_doc[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfull_text\u001b[39m\u001b[38;5;124m\"\u001b[39m][:\u001b[38;5;241m1000\u001b[39m],\n\u001b[0;32m     14\u001b[0m     metadata\u001b[38;5;241m=\u001b[39m{\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124msource\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mearnings_call\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdate\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m2024-Q1\u001b[39m\u001b[38;5;124m\"\u001b[39m},\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     17\u001b[0m     link_entities\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[0;32m     18\u001b[0m )\n\u001b[0;32m     20\u001b[0m results \u001b[38;5;241m=\u001b[39m agent_context\u001b[38;5;241m.\u001b[39mretrieve(\n\u001b[0;32m     21\u001b[0m     query\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mWhat was discussed about revenue growth?\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m     22\u001b[0m     max_results\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m5\u001b[39m,\n\u001b[0;32m     23\u001b[0m     expand_graph\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[0;32m     24\u001b[0m     include_entities\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[0;32m     25\u001b[0m )\n",
      "\u001b[1;31mNameError\u001b[0m: name 'vector_store' is not defined"
     ]
    }
   ],
   "source": [
    "from semantica.context import AgentContext\n",
    "\n",
    "agent_context = AgentContext(\n",
    "    vector_store=vector_store,\n",
    "    knowledge_graph=knowledge_graph,\n",
    "    use_graph_expansion=True,\n",
    "    max_expansion_hops=2,\n",
    "    hybrid_alpha=0.6,\n",
    "    retention_days=30,\n",
    ")\n",
    "\n",
    "memory_id = agent_context.store(\n",
    "    content=chunks,\n",
    "    metadata={\"source\": \"earnings_call\", \"date\": \"2024-Q1\"},\n",
    "    extract_entities=True,\n",
    "    extract_relationships=True,\n",
    "    link_entities=True,\n",
    ")\n",
    "\n",
    "results = agent_context.retrieve(\n",
    "    query=\"What was discussed about revenue growth?\",\n",
    "    max_results=5,\n",
    "    expand_graph=True,\n",
    "    include_entities=True,\n",
    ")\n",
    "\n",
    "stats = agent_context.stats()\n",
    "\n",
    "print(\"AgentContext configured\")\n",
    "print(\"Memory stored:\", memory_id)\n",
    "print(\"GraphRAG results:\", len(results))\n",
    "print(\"Total memories:\", stats.get(\"total_memories\", 0))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 15: Answer Generation\n",
    "\n",
    "Generate answers to financial questions using Groq LLM with retrieved context and knowledge graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "financial_questions = [\n",
    "    \"What were the key financial metrics discussed?\",\n",
    "    \"What guidance was provided for future quarters?\",\n",
    "]\n",
    "\n",
    "generated_answers = []\n",
    "\n",
    "print(\"--- Generating Enhanced Answers ---\\n\")\n",
    "\n",
    "def format_context(retrieved_contexts):\n",
    "    \"\"\"Formats retrieved context with graph information.\"\"\"\n",
    "    formatted_parts = []\n",
    "    \n",
    "    for i, ctx in enumerate(retrieved_contexts):\n",
    "        content = getattr(ctx, \"content\", \"\")\n",
    "        source = getattr(ctx, \"source\", \"unknown\")\n",
    "        \n",
    "        # Format related entities from the graph\n",
    "        related_entities = getattr(ctx, \"related_entities\", [])\n",
    "        entities_str = \", \".join([\n",
    "            f\"{e.get('name', 'Unknown')} ({e.get('type', 'Entity')})\" \n",
    "            for e in related_entities[:5]  # Limit to top 5 per chunk\n",
    "        ])\n",
    "        \n",
    "        # Format related relationships\n",
    "        related_rels = getattr(ctx, \"related_relationships\", [])\n",
    "        rels_str = \"; \".join([\n",
    "            f\"{r.get('source', '')} -> {r.get('type', '')} -> {r.get('target', '')}\"\n",
    "            for r in related_rels[:3]  # Limit to top 3 per chunk\n",
    "        ])\n",
    "        \n",
    "        part = f\"Source {i+1} ({source}):\\n{content}\\n\"\n",
    "        if entities_str:\n",
    "            part += f\"Related Entities: {entities_str}\\n\"\n",
    "        if rels_str:\n",
    "            part += f\"Graph Connections: {rels_str}\\n\"\n",
    "            \n",
    "        formatted_parts.append(part)\n",
    "        \n",
    "    return \"\\n---\\n\".join(formatted_parts)\n",
    "\n",
    "for question in financial_questions:\n",
    "    print(f\"Question: {question}\")\n",
    "    \n",
    "    # Retrieve with graph expansion enabled and higher limits\n",
    "    retrieved_contexts = context_retriever.retrieve(\n",
    "        query=question,\n",
    "        max_results=10,          # Increased from 3\n",
    "        min_relevance_score=0.2,\n",
    "        use_graph_expansion=True, # Explicitly enable graph expansion\n",
    "        max_hops=2               # Traverse up to 2 hops in the graph\n",
    "    )\n",
    "\n",
    "    # Use the rich formatter\n",
    "    context_text = format_context(retrieved_contexts)\n",
    "\n",
    "    # Get global key entities (optional, but good for high-level context)\n",
    "    global_entities = [\n",
    "        f\"{e.get('name', '')} ({e.get('type', '')})\"\n",
    "        for e in knowledge_graph.get(\"entities\", [])[:10]\n",
    "    ]\n",
    "    global_entities_text = \", \".join(global_entities)\n",
    "\n",
    "    prompt = f\"\"\"\n",
    "Answer the question comprehensively using the provided context.\n",
    "The context includes text chunks and knowledge graph connections (entities and relationships).\n",
    "If the answer is not present, say so.\n",
    "\n",
    "Context:\n",
    "{context_text}\n",
    "\n",
    "Global Key Entities: {global_entities_text}\n",
    "\n",
    "Question:\n",
    "{question}\n",
    "\n",
    "Answer:\n",
    "\"\"\".strip()\n",
    "\n",
    "    try:\n",
    "        answer = groq_llm.generate(\n",
    "            prompt,\n",
    "            temperature=0.3, # Lower temperature for more factual answers\n",
    "            max_tokens=1000, # Allow longer answers\n",
    "        )\n",
    "    except Exception as error:\n",
    "        answer = f\"Answer generation failed: {error}\"\n",
    "\n",
    "    generated_answers.append(answer)\n",
    "    print(f\"Answer: {answer}\\n\")\n",
    "    print(\"-\" * 50 + \"\\n\")\n",
    "\n",
    "print(\"Answer generation completed\")\n",
    "print(\"Questions answered:\", len(generated_answers))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 16: Export Results\n",
    "\n",
    "Export knowledge graph and analysis results to JSON and RDF formats.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import JSONExporter, RDFExporter\n",
    "import json\n",
    "\n",
    "# Initialize exporters\n",
    "json_exporter = JSONExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "\n",
    "# Define output file paths\n",
    "json_output_path = \"knowledge_graph.json\"\n",
    "rdf_output_path = \"knowledge_graph.ttl\"\n",
    "\n",
    "# Export to files (required by the API)\n",
    "json_exporter.export(knowledge_graph, file_path=json_output_path, format=\"json\")\n",
    "\n",
    "# FIXED: Use .export() instead of .export_to_rdf() to write to disk\n",
    "rdf_exporter.export(knowledge_graph, file_path=rdf_output_path, format=\"turtle\")\n",
    "\n",
    "# Load the RDF file content to check its size\n",
    "with open(rdf_output_path, \"r\", encoding=\"utf-8\") as f:\n",
    "    kg_rdf_content = f.read()\n",
    "\n",
    "# Create analysis summary\n",
    "analysis_summary = {\n",
    "    \"entities\": len(knowledge_graph.get(\"entities\", [])),\n",
    "    \"relationships\": len(knowledge_graph.get(\"relationships\", [])),\n",
    "    \"entity_conflicts_resolved\": len(locals().get(\"resolved_entity_value_conflicts\", [])),\n",
    "    \"relationship_conflicts_resolved\": len(locals().get(\"resolved_relationship_conflicts\", [])),\n",
    "    \"deduplicated_entities\": len(locals().get(\"deduplicated_entities\", [])),\n",
    "    \"communities\": locals().get(\"num_communities\", 0),\n",
    "    \"questions_answered\": len(generated_answers),\n",
    "    \"llm_model\": getattr(groq_llm, \"model\", \"unknown\"),\n",
    "}\n",
    "\n",
    "print(\"Export completed\")\n",
    "print(\"KG JSON entities:\", analysis_summary[\"entities\"])\n",
    "print(\"KG RDF size (chars):\", len(kg_rdf_content))\n",
    "print(\"Questions answered:\", analysis_summary[\"questions_answered\"])\n",
    "print(\"LLM model:\", analysis_summary[\"llm_model\"])\n",
    "print(\"Conflicts resolved:\", analysis_summary[\"entity_conflicts_resolved\"] + analysis_summary[\"relationship_conflicts_resolved\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
