{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Graph Analytics \n", "\n", "Welcome to the **comprehensive walkthrough** of Semantica's Graph Analytics capabilities. This notebook goes beyond simple graph construction to demonstrate a full-lifecycle production pipeline.\n", "\n", "We will simulate a messy, real-world scenario involving a **Startup Ecosystem** (Investors, Startups, Founders) and guide you through every step of the process:\n", "\n", "1. **Validation**: Catching bad data before it enters the graph.\n", "2. **Cleaning**: Deduplicating entities and resolving conflicts.\n", "3. **Structural Analysis**: Understanding the shape and health of your network.\n", "4. **Deep Analytics**: Centrality, Communities, and Path Finding.\n", "5. **Temporal Analytics**: Time-traveling through your graph data.\n", "6. **Provenance**: Tracking where your data came from.\n", "\n", "Let's dive in!" ] }, { "cell_type": "code", "execution_count": null, "id": "695d435c", "metadata": {}, "outputs": [], "source": [ "!pip install -q semantica" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import logging\n", "import json\n", "from datetime import datetime\n", "\n", "# Set up logging to see what's happening under the hood\n", "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n", "\n", "# Import all the powerful tools from Semantica\n", "from semantica.kg import (\n", " GraphBuilder,\n", " GraphAnalyzer,\n", " GraphValidator,\n", " ConnectivityAnalyzer,\n", " CentralityCalculator,\n", " CommunityDetector,\n", " TemporalGraphQuery,\n", " ProvenanceTracker\n", ")\n", "from semantica.deduplication import DuplicateDetector\n", "from semantica.conflicts import ConflictDetector, ConflictResolver" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. The Scenario: A Messy Startup Ecosystem\n", "\n", "We have data from multiple sources (scrapers, news, user submissions). It's messy:\n", "- **Duplicates**: \"TechFlow AI\" and \"TechFlow Inc.\"\n", "- **Conflicts**: Different revenue numbers for the same company.\n", "- **Errors**: Relationships pointing to non-existent nodes (dangling edges).\n", "- **History**: Investment rounds happening at different times." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Our \"Raw\" Messy Data\n", "raw_entities = [\n", " {\"id\": \"startup_1\", \"type\": \"Startup\", \"name\": \"TechFlow AI\", \"revenue\": 1000000, \"founded\": \"2021-01-01\"},\n", " {\"id\": \"startup_2\", \"type\": \"Startup\", \"name\": \"GreenEnergy Co\", \"revenue\": 500000, \"founded\": \"2020-05-15\"},\n", " {\"id\": \"startup_1_dup\", \"type\": \"Startup\", \"name\": \"TechFlow Inc.\", \"revenue\": 1200000, \"founded\": \"2021-01-01\"}, # Duplicate!\n", " {\"id\": \"investor_1\", \"type\": \"Investor\", \"name\": \"Venture Capital X\"},\n", " {\"id\": \"founder_1\", \"type\": \"Person\", \"name\": \"Alice Chen\"},\n", " {\"id\": \"founder_2\", \"type\": \"Person\", \"name\": \"Bob Smith\"}\n", "]\n", "\n", "raw_relationships = [\n", " # Valid Relationships\n", " {\"source\": \"founder_1\", \"target\": \"startup_1\", \"type\": \"FOUNDED\", \"valid_from\": \"2021-01-01\"},\n", " {\"source\": \"investor_1\", \"target\": \"startup_1\", \"type\": \"INVESTED_IN\", \"amount\": 5000000, \"valid_from\": \"2023-06-01\"},\n", " \n", " # Dangling Edge (Error!)\n", " {\"source\": \"founder_2\", \"target\": \"startup_999\", \"type\": \"FOUNDED\", \"valid_from\": \"2020-05-15\"}, \n", " \n", " # Temporal Data (History)\n", " {\"source\": \"founder_1\", \"target\": \"startup_2\", \"type\": \"ADVISED\", \"valid_from\": \"2020-01-01\", \"valid_until\": \"2021-01-01\"}\n", "]\n", "\n", "print(f\"Loaded {len(raw_entities)} raw entities and {len(raw_relationships)} raw relationships.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Phase 1: Validation (The Gatekeeper)\n", "\n", "Before we do anything, we must validate the graph. Bad data in = Bad insights out.\n", "We use `GraphValidator` to check for:\n", "- **Structural Integrity**: Are all relationship targets present?\n", "- **Schema Compliance**: Do entities have required fields?\n", "- **Consistency**: Are IDs unique?" ] }, { "cell_type": "code", "execution_count": null, "id": "bd8fb13d", "metadata": {}, "outputs": [], "source": [ "# Initialize Validator\n", "validator = GraphValidator()\n", "\n", "# Create a temporary graph object for validation\n", "temp_graph = {\"entities\": raw_entities, \"relationships\": raw_relationships}\n", "\n", "# Run Validation\n", "print(\"Running Validation Check...\")\n", "validation_result = validator.validate(temp_graph)\n", "\n", "if not validation_result.is_valid:\n", " print(\"Validation Failed! Issues found:\")\n", " for issue in validation_result.issues:\n", " print(f\" - [{issue.severity.name}] {issue.message} (Code: {issue.code})\")\n", " \n", " # AUTOMATIC FIX: If it's a dangling edge, remove it\n", " if issue.code == \"DANGLING_EDGE\":\n", " print(\" Auto-Fixing: Removing invalid relationship...\")\n", " raw_relationships = [r for r in raw_relationships \n", " if r['target'] != issue.details.get('target_id')]\n", "else:\n", " print(\"Graph is valid!\")\n", "\n", "# Re-validate to confirm fix\n", "print(\"\\nRe-validating after fixes...\")\n", "temp_graph = {\"entities\": raw_entities, \"relationships\": raw_relationships}\n", "if validator.validate(temp_graph).is_valid:\n", " print(\"Graph is now clean and valid!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Phase 2: Deduplication & Conflict Resolution\n", "\n", "We have \"TechFlow AI\" and \"TechFlow Inc.\". These are likely the same company.\n", "We also have conflicting revenue data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 1. Detect Duplicates\n", "print(\"Scanning for duplicates...\")\n", "deduper = DuplicateDetector(similarity_threshold=0.7) # 70% similarity threshold\n", "duplicates = deduper.detect_duplicates(raw_entities)\n", "\n", "for candidate in duplicates:\n", " print(f\"Found potential duplicate pair (Score: {candidate.similarity_score:.2f}):\")\n", " print(f\" - {candidate.entity1['name']} (ID: {candidate.entity1['id']})\")\n", " print(f\" - {candidate.entity2['name']} (ID: {candidate.entity2['id']})\")\n", " \n", " # MERGE STRATEGY: Keep entity1, merge data from entity2\n", " print(\" Merging entities...\")\n", " # (In a real app, you'd use EntityMerger, but here's the logic:)\n", " # We keep startup_1 and discard startup_1_dup, but we note the conflict\n", " \n", "# 2. Detect Conflicts\n", "print(\"\\nChecking for data conflicts...\")\n", "conflict_detector = ConflictDetector()\n", "\n", "# Simulating a conflict check between the two versions of TechFlow\n", "# To check conflicts, we treat them as the same entity (same ID)\n", "entity_a = raw_entities[0].copy()\n", "entity_b = raw_entities[2].copy()\n", "entity_b['id'] = entity_a['id'] # Force same ID for conflict detection\n", "\n", "conflicts = conflict_detector.detect_conflicts([entity_a, entity_b])\n", "\n", "for conflict in conflicts:\n", " print(f\" Conflict detected in field '{conflict.property_name}':\")\n", " print(f\" Values: {conflict.conflicting_values}\")\n", " \n", " # RESOLUTION: Trust the higher number (optimistic!)\n", " if conflict.property_name == \"revenue\":\n", " # values are strings or ints, need to handle types\n", " vals = [float(v) for v in conflict.conflicting_values if v is not None]\n", " resolved_val = max(vals)\n", " print(f\" Resolved to: {resolved_val}\")\n", " raw_entities[0]['revenue'] = resolved_val\n", "\n", "# Final Cleanup: Remove the duplicate entity from our list\n", "clean_entities = [e for e in raw_entities if e['id'] != 'startup_1_dup']\n", "clean_relationships = raw_relationships # (We'd normally re-link relationships too)\n", "\n", "print(f\"\\nCleaned Data: {len(clean_entities)} entities remaining.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Phase 3: Building the Knowledge Graph\n", "\n", "Now that our data is clean, we build the official graph object." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Manual Graph Construction (since we already cleaned it)\n", "kg = {\n", " \"entities\": clean_entities,\n", " \"relationships\": clean_relationships,\n", " \"metadata\": {\n", " \"created_at\": datetime.now().isoformat(),\n", " \"source\": \"Manual Advanced Pipeline\"\n", " }\n", "}\n", "print(\"Knowledge Graph Assembled Successfully!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Phase 4: Advanced Analytics\n", "\n", "This is where the magic happens. We'll use multiple analyzers to extract insights." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Initialize the Master Analyzer\n", "analyzer = GraphAnalyzer(enable_temporal=True)\n", "\n", "# 1. Structural Analysis (Connectivity)\n", "print(\"\\n--- Connectivity Analysis ---\")\n", "connectivity = analyzer.analyze_connectivity(kg)\n", "print(f\" • Graph Connected? {'Yes' if connectivity['is_connected'] else 'No'}\")\n", "print(f\" • Connected Components: {connectivity['num_components']}\")\n", "\n", "# 2. Centrality (Who is important?)\n", "print(\"\\n--- Centrality Analysis ---\")\n", "centrality_result = analyzer.calculate_centrality(kg, centrality_type=\"degree\")\n", "degree_data = centrality_result[\"centrality_measures\"][\"degree\"]\n", "\n", "# Get pre-calculated rankings\n", "top_nodes = degree_data[\"rankings\"][:3]\n", "\n", "print(\" • Top Influencers (Degree Centrality):\")\n", "for item in top_nodes:\n", " print(f\" - {item['node']}: {item['score']:.2f}\")\n", "\n", "# 3. Community Detection (Clustering)\n", "print(\"\\n--- Community Detection ---\")\n", "community_result = analyzer.detect_communities(kg, algorithm=\"louvain\")\n", "communities = community_result[\"communities\"]\n", "\n", "print(f\" • Detected {len(communities)} communities.\")\n", "for i, comm in enumerate(communities):\n", " # comm is a set of node IDs\n", " members = list(comm)\n", " print(f\" Community {i+1}: {', '.join(members)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Phase 5: Temporal Analytics (Time Travel)\n", "\n", "Static graphs are boring. Real worlds change. Let's analyze the **evolution** of our ecosystem." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temporal_engine = TemporalGraphQuery(temporal_granularity=\"year\")\n", "\n", "# 1. Time Travel Query: What did the world look like in 2020?\n", "print(\"\\n--- Time Travel: 2020 ---\")\n", "snapshot_2020 = temporal_engine.query_at_time(kg, query=\"*\", at_time=\"2020-06-01\")\n", "print(f\" Active Relationships in 2020: {len(snapshot_2020['relationships'])}\")\n", "for rel in snapshot_2020['relationships']:\n", " print(f\" - {rel['source']} --[{rel['type']}]--> {rel['target']}\")\n", "\n", "# 2. Time Travel Query: What about 2023?\n", "print(\"\\n--- Time Travel: 2023 ---\")\n", "snapshot_2023 = temporal_engine.query_at_time(kg, query=\"*\", at_time=\"2023-07-01\")\n", "print(f\" Active Relationships in 2023: {len(snapshot_2023['relationships'])}\")\n", "for rel in snapshot_2023['relationships']:\n", " print(f\" - {rel['source']} --[{rel['type']}]--> {rel['target']}\")\n", " \n", "# Notice how 'ADVISED' might disappear if it ended, and 'INVESTED_IN' appears!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Phase 6: Provenance (Data Lineage)\n", "\n", "Finally, in a production system, you need to know **where** a fact came from. This is crucial for trust." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tracker = ProvenanceTracker()\n", "\n", "# Let's pretend we're tracking the source of our data\n", "tracker.track_entity(\"startup_1\", source=\"Crunchbase_API_v2\", metadata={\"confidence\": 0.95})\n", "tracker.track_entity(\"startup_1\", source=\"Manual_Entry_User_Bob\", metadata={\"confidence\": 1.0})\n", "\n", "print(\"\\n--- Provenance Report: TechFlow AI ---\")\n", "lineage = tracker.get_lineage(\"startup_1\")\n", "print(f\" Entity: startup_1\")\n", "print(f\" First Seen: {lineage['first_seen']}\")\n", "print(f\" Sources:\")\n", "for src in lineage['sources']:\n", " print(f\" - {src['source']} (at {src['timestamp']})\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "You have just walked through a complete, advanced Knowledge Graph pipeline:\n", "\n", "1. **Validated** messy input data.\n", "2. **Cleaned** duplicates and conflicts.\n", "3. **Analyzed** structure and community dynamics.\n", "4. **Queried** across time dimensions.\n", "5. **Tracked** data lineage.\n", "\n", "This represents the state-of-the-art in modern KG Engineering using Semantica." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }