{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Graph Analytics \n",
    "\n",
    "Welcome to the **comprehensive walkthrough** of Semantica's Graph Analytics capabilities. This notebook goes beyond simple graph construction to demonstrate a full-lifecycle production pipeline.\n",
    "\n",
    "We will simulate a messy, real-world scenario involving a **Startup Ecosystem** (Investors, Startups, Founders) and guide you through every step of the process:\n",
    "\n",
    "1.  **Validation**: Catching bad data before it enters the graph.\n",
    "2.  **Cleaning**: Deduplicating entities and resolving conflicts.\n",
    "3.  **Structural Analysis**: Understanding the shape and health of your network.\n",
    "4.  **Deep Analytics**: Centrality, Communities, and Path Finding.\n",
    "5.  **Temporal Analytics**: Time-traveling through your graph data.\n",
    "6.  **Provenance**: Tracking where your data came from.\n",
    "\n",
    "Let's dive in!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "695d435c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q semantica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import json\n",
    "from datetime import datetime\n",
    "\n",
    "# Set up logging to see what's happening under the hood\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "# Import all the powerful tools from Semantica\n",
    "from semantica.kg import (\n",
    "    GraphBuilder,\n",
    "    GraphAnalyzer,\n",
    "    GraphValidator,\n",
    "    ConnectivityAnalyzer,\n",
    "    CentralityCalculator,\n",
    "    CommunityDetector,\n",
    "    TemporalGraphQuery,\n",
    "    ProvenanceTracker\n",
    ")\n",
    "from semantica.deduplication import DuplicateDetector\n",
    "from semantica.conflicts import ConflictDetector, ConflictResolver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. The Scenario: A Messy Startup Ecosystem\n",
    "\n",
    "We have data from multiple sources (scrapers, news, user submissions). It's messy:\n",
    "- **Duplicates**: \"TechFlow AI\" and \"TechFlow Inc.\"\n",
    "- **Conflicts**: Different revenue numbers for the same company.\n",
    "- **Errors**: Relationships pointing to non-existent nodes (dangling edges).\n",
    "- **History**: Investment rounds happening at different times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our \"Raw\" Messy Data\n",
    "raw_entities = [\n",
    "    {\"id\": \"startup_1\", \"type\": \"Startup\", \"name\": \"TechFlow AI\", \"revenue\": 1000000, \"founded\": \"2021-01-01\"},\n",
    "    {\"id\": \"startup_2\", \"type\": \"Startup\", \"name\": \"GreenEnergy Co\", \"revenue\": 500000, \"founded\": \"2020-05-15\"},\n",
    "    {\"id\": \"startup_1_dup\", \"type\": \"Startup\", \"name\": \"TechFlow Inc.\", \"revenue\": 1200000, \"founded\": \"2021-01-01\"}, # Duplicate!\n",
    "    {\"id\": \"investor_1\", \"type\": \"Investor\", \"name\": \"Venture Capital X\"},\n",
    "    {\"id\": \"founder_1\", \"type\": \"Person\", \"name\": \"Alice Chen\"},\n",
    "    {\"id\": \"founder_2\", \"type\": \"Person\", \"name\": \"Bob Smith\"}\n",
    "]\n",
    "\n",
    "raw_relationships = [\n",
    "    # Valid Relationships\n",
    "    {\"source\": \"founder_1\", \"target\": \"startup_1\", \"type\": \"FOUNDED\", \"valid_from\": \"2021-01-01\"},\n",
    "    {\"source\": \"investor_1\", \"target\": \"startup_1\", \"type\": \"INVESTED_IN\", \"amount\": 5000000, \"valid_from\": \"2023-06-01\"},\n",
    "    \n",
    "    # Dangling Edge (Error!)\n",
    "    {\"source\": \"founder_2\", \"target\": \"startup_999\", \"type\": \"FOUNDED\", \"valid_from\": \"2020-05-15\"}, \n",
    "    \n",
    "    # Temporal Data (History)\n",
    "    {\"source\": \"founder_1\", \"target\": \"startup_2\", \"type\": \"ADVISED\", \"valid_from\": \"2020-01-01\", \"valid_until\": \"2021-01-01\"}\n",
    "]\n",
    "\n",
    "print(f\"Loaded {len(raw_entities)} raw entities and {len(raw_relationships)} raw relationships.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Phase 1: Validation (The Gatekeeper)\n",
    "\n",
    "Before we do anything, we must validate the graph. Bad data in = Bad insights out.\n",
    "We use `GraphValidator` to check for:\n",
    "- **Structural Integrity**: Are all relationship targets present?\n",
    "- **Schema Compliance**: Do entities have required fields?\n",
    "- **Consistency**: Are IDs unique?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd8fb13d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Validator\n",
    "validator = GraphValidator()\n",
    "\n",
    "# Create a temporary graph object for validation\n",
    "temp_graph = {\"entities\": raw_entities, \"relationships\": raw_relationships}\n",
    "\n",
    "# Run Validation\n",
    "print(\"Running Validation Check...\")\n",
    "validation_result = validator.validate(temp_graph)\n",
    "\n",
    "if not validation_result.is_valid:\n",
    "    print(\"Validation Failed! Issues found:\")\n",
    "    for issue in validation_result.issues:\n",
    "        print(f\"   - [{issue.severity.name}] {issue.message} (Code: {issue.code})\")\n",
    "        \n",
    "        # AUTOMATIC FIX: If it's a dangling edge, remove it\n",
    "        if issue.code == \"DANGLING_EDGE\":\n",
    "            print(\"     Auto-Fixing: Removing invalid relationship...\")\n",
    "            raw_relationships = [r for r in raw_relationships \n",
    "                               if r['target'] != issue.details.get('target_id')]\n",
    "else:\n",
    "    print(\"Graph is valid!\")\n",
    "\n",
    "# Re-validate to confirm fix\n",
    "print(\"\\nRe-validating after fixes...\")\n",
    "temp_graph = {\"entities\": raw_entities, \"relationships\": raw_relationships}\n",
    "if validator.validate(temp_graph).is_valid:\n",
    "    print(\"Graph is now clean and valid!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Phase 2: Deduplication & Conflict Resolution\n",
    "\n",
    "We have \"TechFlow AI\" and \"TechFlow Inc.\". These are likely the same company.\n",
    "We also have conflicting revenue data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Detect Duplicates\n",
    "print(\"Scanning for duplicates...\")\n",
    "deduper = DuplicateDetector(similarity_threshold=0.7) # 70% similarity threshold\n",
    "duplicates = deduper.detect_duplicates(raw_entities)\n",
    "\n",
    "for candidate in duplicates:\n",
    "    print(f\"Found potential duplicate pair (Score: {candidate.similarity_score:.2f}):\")\n",
    "    print(f\"   - {candidate.entity1['name']} (ID: {candidate.entity1['id']})\")\n",
    "    print(f\"   - {candidate.entity2['name']} (ID: {candidate.entity2['id']})\")\n",
    "    \n",
    "    # MERGE STRATEGY: Keep entity1, merge data from entity2\n",
    "    print(\"   Merging entities...\")\n",
    "    # (In a real app, you'd use EntityMerger, but here's the logic:)\n",
    "    # We keep startup_1 and discard startup_1_dup, but we note the conflict\n",
    "    \n",
    "# 2. Detect Conflicts\n",
    "print(\"\\nChecking for data conflicts...\")\n",
    "conflict_detector = ConflictDetector()\n",
    "\n",
    "# Simulating a conflict check between the two versions of TechFlow\n",
    "# To check conflicts, we treat them as the same entity (same ID)\n",
    "entity_a = raw_entities[0].copy()\n",
    "entity_b = raw_entities[2].copy()\n",
    "entity_b['id'] = entity_a['id'] # Force same ID for conflict detection\n",
    "\n",
    "conflicts = conflict_detector.detect_conflicts([entity_a, entity_b])\n",
    "\n",
    "for conflict in conflicts:\n",
    "    print(f\"   Conflict detected in field '{conflict.property_name}':\")\n",
    "    print(f\"      Values: {conflict.conflicting_values}\")\n",
    "    \n",
    "    # RESOLUTION: Trust the higher number (optimistic!)\n",
    "    if conflict.property_name == \"revenue\":\n",
    "        # values are strings or ints, need to handle types\n",
    "        vals = [float(v) for v in conflict.conflicting_values if v is not None]\n",
    "        resolved_val = max(vals)\n",
    "        print(f\"      Resolved to: {resolved_val}\")\n",
    "        raw_entities[0]['revenue'] = resolved_val\n",
    "\n",
    "# Final Cleanup: Remove the duplicate entity from our list\n",
    "clean_entities = [e for e in raw_entities if e['id'] != 'startup_1_dup']\n",
    "clean_relationships = raw_relationships # (We'd normally re-link relationships too)\n",
    "\n",
    "print(f\"\\nCleaned Data: {len(clean_entities)} entities remaining.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Phase 3: Building the Knowledge Graph\n",
    "\n",
    "Now that our data is clean, we build the official graph object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Manual Graph Construction (since we already cleaned it)\n",
    "kg = {\n",
    "    \"entities\": clean_entities,\n",
    "    \"relationships\": clean_relationships,\n",
    "    \"metadata\": {\n",
    "        \"created_at\": datetime.now().isoformat(),\n",
    "        \"source\": \"Manual Advanced Pipeline\"\n",
    "    }\n",
    "}\n",
    "print(\"Knowledge Graph Assembled Successfully!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Phase 4: Advanced Analytics\n",
    "\n",
    "This is where the magic happens. We'll use multiple analyzers to extract insights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Master Analyzer\n",
    "analyzer = GraphAnalyzer(enable_temporal=True)\n",
    "\n",
    "# 1. Structural Analysis (Connectivity)\n",
    "print(\"\\n--- Connectivity Analysis ---\")\n",
    "connectivity = analyzer.analyze_connectivity(kg)\n",
    "print(f\"   • Graph Connected? {'Yes' if connectivity['is_connected'] else 'No'}\")\n",
    "print(f\"   • Connected Components: {connectivity['num_components']}\")\n",
    "\n",
    "# 2. Centrality (Who is important?)\n",
    "print(\"\\n--- Centrality Analysis ---\")\n",
    "centrality_result = analyzer.calculate_centrality(kg, centrality_type=\"degree\")\n",
    "degree_data = centrality_result[\"centrality_measures\"][\"degree\"]\n",
    "\n",
    "# Get pre-calculated rankings\n",
    "top_nodes = degree_data[\"rankings\"][:3]\n",
    "\n",
    "print(\"   • Top Influencers (Degree Centrality):\")\n",
    "for item in top_nodes:\n",
    "    print(f\"     - {item['node']}: {item['score']:.2f}\")\n",
    "\n",
    "# 3. Community Detection (Clustering)\n",
    "print(\"\\n--- Community Detection ---\")\n",
    "community_result = analyzer.detect_communities(kg, algorithm=\"louvain\")\n",
    "communities = community_result[\"communities\"]\n",
    "\n",
    "print(f\"   • Detected {len(communities)} communities.\")\n",
    "for i, comm in enumerate(communities):\n",
    "    # comm is a set of node IDs\n",
    "    members = list(comm)\n",
    "    print(f\"     Community {i+1}: {', '.join(members)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Phase 5: Temporal Analytics (Time Travel)\n",
    "\n",
    "Static graphs are boring. Real worlds change. Let's analyze the **evolution** of our ecosystem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temporal_engine = TemporalGraphQuery(temporal_granularity=\"year\")\n",
    "\n",
    "# 1. Time Travel Query: What did the world look like in 2020?\n",
    "print(\"\\n--- Time Travel: 2020 ---\")\n",
    "snapshot_2020 = temporal_engine.query_at_time(kg, query=\"*\", at_time=\"2020-06-01\")\n",
    "print(f\"   Active Relationships in 2020: {len(snapshot_2020['relationships'])}\")\n",
    "for rel in snapshot_2020['relationships']:\n",
    "    print(f\"   - {rel['source']} --[{rel['type']}]--> {rel['target']}\")\n",
    "\n",
    "# 2. Time Travel Query: What about 2023?\n",
    "print(\"\\n--- Time Travel: 2023 ---\")\n",
    "snapshot_2023 = temporal_engine.query_at_time(kg, query=\"*\", at_time=\"2023-07-01\")\n",
    "print(f\"   Active Relationships in 2023: {len(snapshot_2023['relationships'])}\")\n",
    "for rel in snapshot_2023['relationships']:\n",
    "    print(f\"   - {rel['source']} --[{rel['type']}]--> {rel['target']}\")\n",
    "    \n",
    "# Notice how 'ADVISED' might disappear if it ended, and 'INVESTED_IN' appears!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Phase 6: Provenance (Data Lineage)\n",
    "\n",
    "Finally, in a production system, you need to know **where** a fact came from. This is crucial for trust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = ProvenanceTracker()\n",
    "\n",
    "# Let's pretend we're tracking the source of our data\n",
    "tracker.track_entity(\"startup_1\", source=\"Crunchbase_API_v2\", metadata={\"confidence\": 0.95})\n",
    "tracker.track_entity(\"startup_1\", source=\"Manual_Entry_User_Bob\", metadata={\"confidence\": 1.0})\n",
    "\n",
    "print(\"\\n--- Provenance Report: TechFlow AI ---\")\n",
    "lineage = tracker.get_lineage(\"startup_1\")\n",
    "print(f\"   Entity: startup_1\")\n",
    "print(f\"   First Seen: {lineage['first_seen']}\")\n",
    "print(f\"   Sources:\")\n",
    "for src in lineage['sources']:\n",
    "    print(f\"     - {src['source']} (at {src['timestamp']})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "You have just walked through a complete, advanced Knowledge Graph pipeline:\n",
    "\n",
    "1.  **Validated** messy input data.\n",
    "2.  **Cleaned** duplicates and conflicts.\n",
    "3.  **Analyzed** structure and community dynamics.\n",
    "4.  **Queried** across time dimensions.\n",
    "5.  **Tracked** data lineage.\n",
    "\n",
    "This represents the state-of-the-art in modern KG Engineering using Semantica."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
