mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-09 04:00:52 +00:00
396 lines
15 KiB
Plaintext
396 lines
15 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Graph Analytics \n",
|
|
"\n",
|
|
"Welcome to the **comprehensive walkthrough** of Semantica's Graph Analytics capabilities. This notebook goes beyond simple graph construction to demonstrate a full-lifecycle production pipeline.\n",
|
|
"\n",
|
|
"We will simulate a messy, real-world scenario involving a **Startup Ecosystem** (Investors, Startups, Founders) and guide you through every step of the process:\n",
|
|
"\n",
|
|
"1. **Validation**: Catching bad data before it enters the graph.\n",
|
|
"2. **Cleaning**: Deduplicating entities and resolving conflicts.\n",
|
|
"3. **Structural Analysis**: Understanding the shape and health of your network.\n",
|
|
"4. **Deep Analytics**: Centrality, Communities, and Path Finding.\n",
|
|
"5. **Temporal Analytics**: Time-traveling through your graph data.\n",
|
|
"6. **Provenance**: Tracking where your data came from.\n",
|
|
"\n",
|
|
"Let's dive in!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "695d435c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install -q semantica"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import logging\n",
|
|
"import json\n",
|
|
"from datetime import datetime\n",
|
|
"\n",
|
|
"# Set up logging to see what's happening under the hood\n",
|
|
"logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
|
|
"\n",
|
|
"# Import all the powerful tools from Semantica\n",
|
|
"from semantica.kg import (\n",
|
|
" GraphBuilder,\n",
|
|
" GraphAnalyzer,\n",
|
|
" GraphValidator,\n",
|
|
" ConnectivityAnalyzer,\n",
|
|
" CentralityCalculator,\n",
|
|
" CommunityDetector,\n",
|
|
" TemporalGraphQuery,\n",
|
|
" ProvenanceTracker\n",
|
|
")\n",
|
|
"from semantica.deduplication import DuplicateDetector\n",
|
|
"from semantica.conflicts import ConflictDetector, ConflictResolver"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. The Scenario: A Messy Startup Ecosystem\n",
|
|
"\n",
|
|
"We have data from multiple sources (scrapers, news, user submissions). It's messy:\n",
|
|
"- **Duplicates**: \"TechFlow AI\" and \"TechFlow Inc.\"\n",
|
|
"- **Conflicts**: Different revenue numbers for the same company.\n",
|
|
"- **Errors**: Relationships pointing to non-existent nodes (dangling edges).\n",
|
|
"- **History**: Investment rounds happening at different times."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Our \"Raw\" Messy Data\n",
|
|
"raw_entities = [\n",
|
|
" {\"id\": \"startup_1\", \"type\": \"Startup\", \"name\": \"TechFlow AI\", \"revenue\": 1000000, \"founded\": \"2021-01-01\"},\n",
|
|
" {\"id\": \"startup_2\", \"type\": \"Startup\", \"name\": \"GreenEnergy Co\", \"revenue\": 500000, \"founded\": \"2020-05-15\"},\n",
|
|
" {\"id\": \"startup_1_dup\", \"type\": \"Startup\", \"name\": \"TechFlow Inc.\", \"revenue\": 1200000, \"founded\": \"2021-01-01\"}, # Duplicate!\n",
|
|
" {\"id\": \"investor_1\", \"type\": \"Investor\", \"name\": \"Venture Capital X\"},\n",
|
|
" {\"id\": \"founder_1\", \"type\": \"Person\", \"name\": \"Alice Chen\"},\n",
|
|
" {\"id\": \"founder_2\", \"type\": \"Person\", \"name\": \"Bob Smith\"}\n",
|
|
"]\n",
|
|
"\n",
|
|
"raw_relationships = [\n",
|
|
" # Valid Relationships\n",
|
|
" {\"source\": \"founder_1\", \"target\": \"startup_1\", \"type\": \"FOUNDED\", \"valid_from\": \"2021-01-01\"},\n",
|
|
" {\"source\": \"investor_1\", \"target\": \"startup_1\", \"type\": \"INVESTED_IN\", \"amount\": 5000000, \"valid_from\": \"2023-06-01\"},\n",
|
|
" \n",
|
|
" # Dangling Edge (Error!)\n",
|
|
" {\"source\": \"founder_2\", \"target\": \"startup_999\", \"type\": \"FOUNDED\", \"valid_from\": \"2020-05-15\"}, \n",
|
|
" \n",
|
|
" # Temporal Data (History)\n",
|
|
" {\"source\": \"founder_1\", \"target\": \"startup_2\", \"type\": \"ADVISED\", \"valid_from\": \"2020-01-01\", \"valid_until\": \"2021-01-01\"}\n",
|
|
"]\n",
|
|
"\n",
|
|
"print(f\"Loaded {len(raw_entities)} raw entities and {len(raw_relationships)} raw relationships.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Phase 1: Validation (The Gatekeeper)\n",
|
|
"\n",
|
|
"Before we do anything, we must validate the graph. Bad data in = Bad insights out.\n",
|
|
"We use `GraphValidator` to check for:\n",
|
|
"- **Structural Integrity**: Are all relationship targets present?\n",
|
|
"- **Schema Compliance**: Do entities have required fields?\n",
|
|
"- **Consistency**: Are IDs unique?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "bd8fb13d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Initialize Validator\n",
|
|
"validator = GraphValidator()\n",
|
|
"\n",
|
|
"# Create a temporary graph object for validation\n",
|
|
"temp_graph = {\"entities\": raw_entities, \"relationships\": raw_relationships}\n",
|
|
"\n",
|
|
"# Run Validation\n",
|
|
"print(\"Running Validation Check...\")\n",
|
|
"validation_result = validator.validate(temp_graph)\n",
|
|
"\n",
|
|
"if not validation_result.is_valid:\n",
|
|
" print(\"Validation Failed! Issues found:\")\n",
|
|
" for issue in validation_result.issues:\n",
|
|
" print(f\" - [{issue.severity.name}] {issue.message} (Code: {issue.code})\")\n",
|
|
" \n",
|
|
" # AUTOMATIC FIX: If it's a dangling edge, remove it\n",
|
|
" if issue.code == \"DANGLING_EDGE\":\n",
|
|
" print(\" Auto-Fixing: Removing invalid relationship...\")\n",
|
|
" raw_relationships = [r for r in raw_relationships \n",
|
|
" if r['target'] != issue.details.get('target_id')]\n",
|
|
"else:\n",
|
|
" print(\"Graph is valid!\")\n",
|
|
"\n",
|
|
"# Re-validate to confirm fix\n",
|
|
"print(\"\\nRe-validating after fixes...\")\n",
|
|
"temp_graph = {\"entities\": raw_entities, \"relationships\": raw_relationships}\n",
|
|
"if validator.validate(temp_graph).is_valid:\n",
|
|
" print(\"Graph is now clean and valid!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Phase 2: Deduplication & Conflict Resolution\n",
|
|
"\n",
|
|
"We have \"TechFlow AI\" and \"TechFlow Inc.\". These are likely the same company.\n",
|
|
"We also have conflicting revenue data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# 1. Detect Duplicates\n",
|
|
"print(\"Scanning for duplicates...\")\n",
|
|
"deduper = DuplicateDetector(similarity_threshold=0.7) # 70% similarity threshold\n",
|
|
"duplicates = deduper.detect_duplicates(raw_entities)\n",
|
|
"\n",
|
|
"for candidate in duplicates:\n",
|
|
" print(f\"Found potential duplicate pair (Score: {candidate.similarity_score:.2f}):\")\n",
|
|
" print(f\" - {candidate.entity1['name']} (ID: {candidate.entity1['id']})\")\n",
|
|
" print(f\" - {candidate.entity2['name']} (ID: {candidate.entity2['id']})\")\n",
|
|
" \n",
|
|
" # MERGE STRATEGY: Keep entity1, merge data from entity2\n",
|
|
" print(\" Merging entities...\")\n",
|
|
" # (In a real app, you'd use EntityMerger, but here's the logic:)\n",
|
|
" # We keep startup_1 and discard startup_1_dup, but we note the conflict\n",
|
|
" \n",
|
|
"# 2. Detect Conflicts\n",
|
|
"print(\"\\nChecking for data conflicts...\")\n",
|
|
"conflict_detector = ConflictDetector()\n",
|
|
"\n",
|
|
"# Simulating a conflict check between the two versions of TechFlow\n",
|
|
"# To check conflicts, we treat them as the same entity (same ID)\n",
|
|
"entity_a = raw_entities[0].copy()\n",
|
|
"entity_b = raw_entities[2].copy()\n",
|
|
"entity_b['id'] = entity_a['id'] # Force same ID for conflict detection\n",
|
|
"\n",
|
|
"conflicts = conflict_detector.detect_conflicts([entity_a, entity_b])\n",
|
|
"\n",
|
|
"for conflict in conflicts:\n",
|
|
" print(f\" Conflict detected in field '{conflict.property_name}':\")\n",
|
|
" print(f\" Values: {conflict.conflicting_values}\")\n",
|
|
" \n",
|
|
" # RESOLUTION: Trust the higher number (optimistic!)\n",
|
|
" if conflict.property_name == \"revenue\":\n",
|
|
" # values are strings or ints, need to handle types\n",
|
|
" vals = [float(v) for v in conflict.conflicting_values if v is not None]\n",
|
|
" resolved_val = max(vals)\n",
|
|
" print(f\" Resolved to: {resolved_val}\")\n",
|
|
" raw_entities[0]['revenue'] = resolved_val\n",
|
|
"\n",
|
|
"# Final Cleanup: Remove the duplicate entity from our list\n",
|
|
"clean_entities = [e for e in raw_entities if e['id'] != 'startup_1_dup']\n",
|
|
"clean_relationships = raw_relationships # (We'd normally re-link relationships too)\n",
|
|
"\n",
|
|
"print(f\"\\nCleaned Data: {len(clean_entities)} entities remaining.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Phase 3: Building the Knowledge Graph\n",
|
|
"\n",
|
|
"Now that our data is clean, we build the official graph object."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Manual Graph Construction (since we already cleaned it)\n",
|
|
"kg = {\n",
|
|
" \"entities\": clean_entities,\n",
|
|
" \"relationships\": clean_relationships,\n",
|
|
" \"metadata\": {\n",
|
|
" \"created_at\": datetime.now().isoformat(),\n",
|
|
" \"source\": \"Manual Advanced Pipeline\"\n",
|
|
" }\n",
|
|
"}\n",
|
|
"print(\"Knowledge Graph Assembled Successfully!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 5. Phase 4: Advanced Analytics\n",
|
|
"\n",
|
|
"This is where the magic happens. We'll use multiple analyzers to extract insights."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Initialize the Master Analyzer\n",
|
|
"analyzer = GraphAnalyzer(enable_temporal=True)\n",
|
|
"\n",
|
|
"# 1. Structural Analysis (Connectivity)\n",
|
|
"print(\"\\n--- Connectivity Analysis ---\")\n",
|
|
"connectivity = analyzer.analyze_connectivity(kg)\n",
|
|
"print(f\" • Graph Connected? {'Yes' if connectivity['is_connected'] else 'No'}\")\n",
|
|
"print(f\" • Connected Components: {connectivity['num_components']}\")\n",
|
|
"\n",
|
|
"# 2. Centrality (Who is important?)\n",
|
|
"print(\"\\n--- Centrality Analysis ---\")\n",
|
|
"centrality_result = analyzer.calculate_centrality(kg, centrality_type=\"degree\")\n",
|
|
"degree_data = centrality_result[\"centrality_measures\"][\"degree\"]\n",
|
|
"\n",
|
|
"# Get pre-calculated rankings\n",
|
|
"top_nodes = degree_data[\"rankings\"][:3]\n",
|
|
"\n",
|
|
"print(\" • Top Influencers (Degree Centrality):\")\n",
|
|
"for item in top_nodes:\n",
|
|
" print(f\" - {item['node']}: {item['score']:.2f}\")\n",
|
|
"\n",
|
|
"# 3. Community Detection (Clustering)\n",
|
|
"print(\"\\n--- Community Detection ---\")\n",
|
|
"community_result = analyzer.detect_communities(kg, algorithm=\"louvain\")\n",
|
|
"communities = community_result[\"communities\"]\n",
|
|
"\n",
|
|
"print(f\" • Detected {len(communities)} communities.\")\n",
|
|
"for i, comm in enumerate(communities):\n",
|
|
" # comm is a set of node IDs\n",
|
|
" members = list(comm)\n",
|
|
" print(f\" Community {i+1}: {', '.join(members)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 6. Phase 5: Temporal Analytics (Time Travel)\n",
|
|
"\n",
|
|
"Static graphs are boring. Real worlds change. Let's analyze the **evolution** of our ecosystem."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"temporal_engine = TemporalGraphQuery(temporal_granularity=\"year\")\n",
|
|
"\n",
|
|
"# 1. Time Travel Query: What did the world look like in 2020?\n",
|
|
"print(\"\\n--- Time Travel: 2020 ---\")\n",
|
|
"snapshot_2020 = temporal_engine.query_at_time(kg, query=\"*\", at_time=\"2020-06-01\")\n",
|
|
"print(f\" Active Relationships in 2020: {len(snapshot_2020['relationships'])}\")\n",
|
|
"for rel in snapshot_2020['relationships']:\n",
|
|
" print(f\" - {rel['source']} --[{rel['type']}]--> {rel['target']}\")\n",
|
|
"\n",
|
|
"# 2. Time Travel Query: What about 2023?\n",
|
|
"print(\"\\n--- Time Travel: 2023 ---\")\n",
|
|
"snapshot_2023 = temporal_engine.query_at_time(kg, query=\"*\", at_time=\"2023-07-01\")\n",
|
|
"print(f\" Active Relationships in 2023: {len(snapshot_2023['relationships'])}\")\n",
|
|
"for rel in snapshot_2023['relationships']:\n",
|
|
" print(f\" - {rel['source']} --[{rel['type']}]--> {rel['target']}\")\n",
|
|
" \n",
|
|
"# Notice how 'ADVISED' might disappear if it ended, and 'INVESTED_IN' appears!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 7. Phase 6: Provenance (Data Lineage)\n",
|
|
"\n",
|
|
"Finally, in a production system, you need to know **where** a fact came from. This is crucial for trust."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"tracker = ProvenanceTracker()\n",
|
|
"\n",
|
|
"# Let's pretend we're tracking the source of our data\n",
|
|
"tracker.track_entity(\"startup_1\", source=\"Crunchbase_API_v2\", metadata={\"confidence\": 0.95})\n",
|
|
"tracker.track_entity(\"startup_1\", source=\"Manual_Entry_User_Bob\", metadata={\"confidence\": 1.0})\n",
|
|
"\n",
|
|
"print(\"\\n--- Provenance Report: TechFlow AI ---\")\n",
|
|
"lineage = tracker.get_lineage(\"startup_1\")\n",
|
|
"print(f\" Entity: startup_1\")\n",
|
|
"print(f\" First Seen: {lineage['first_seen']}\")\n",
|
|
"print(f\" Sources:\")\n",
|
|
"for src in lineage['sources']:\n",
|
|
" print(f\" - {src['source']} (at {src['timestamp']})\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Conclusion\n",
|
|
"\n",
|
|
"You have just walked through a complete, advanced Knowledge Graph pipeline:\n",
|
|
"\n",
|
|
"1. **Validated** messy input data.\n",
|
|
"2. **Cleaned** duplicates and conflicts.\n",
|
|
"3. **Analyzed** structure and community dynamics.\n",
|
|
"4. **Queried** across time dimensions.\n",
|
|
"5. **Tracked** data lineage.\n",
|
|
"\n",
|
|
"This represents the state-of-the-art in modern KG Engineering using Semantica."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|