mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Update Multi-Source Integration notebook: separate install, add MCP, remove Advanced keyword
This commit is contained in:
KaifAhmad1
@@ -4,38 +4,23 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)\n",
|
||||
"# Real-World Multi-Source Integration\n",
|
||||
"\n",
|
||||
"# Multi-Source Data Integration\n",
|
||||
"This cookbook demonstrates a production-grade workflow for integrating data from distinct sources using Semantica.\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"We will simulate a complex Enterprise Knowledge Graph construction scenario for **Nexus AI** by aggregating data from:\n",
|
||||
"\n",
|
||||
"This notebook demonstrates advanced multi-source data integration using multiple ingestion types, entity resolution, conflict detection, and provenance tracking.\n",
|
||||
"1. **Corporate Database (SQLite)**: Financial records and employee counts.\n",
|
||||
"2. **Public Web (HTML)**: News articles and press releases.\n",
|
||||
"3. **Source Code (Markdown)**: Engineering activity and documentation.\n",
|
||||
"4. **Market Data API (JSON)**: Live stock prices and market cap.\n",
|
||||
"5. **Web Search MCP (Tool)**: Live competitor analysis from a search agent.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/ingest/)\n",
|
||||
"\n",
|
||||
"### Learning Objectives\n",
|
||||
"\n",
|
||||
"- Ingest data from multiple sources (files, web, databases, streams, feeds)\n",
|
||||
"- Resolve entities across sources using EntityResolver\n",
|
||||
"- Detect conflicts using ConflictDetector\n",
|
||||
"- Track provenance using ProvenanceTracker\n",
|
||||
"- Integrate data into a unified knowledge graph\n",
|
||||
"\n",
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install Semantica from PyPI:\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"pip install semantica\n",
|
||||
"# Or with all optional dependencies:\n",
|
||||
"pip install semantica[all]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Workflow: Multi-Source Ingestion → Entity Resolution → Conflict Detection → Provenance Tracking → Unified KG\n"
|
||||
"**Key Semantica Modules Used:**\n",
|
||||
"* `ingest`: For loading data from disparate sources (including MCP).\n",
|
||||
"* `kg.GraphBuilder`: For constructing the graph and merging entities.\n",
|
||||
"* `conflicts.ConflictResolver`: For resolving data discrepancies.\n",
|
||||
"* `visualization.KGVisualizer`: For visualizing the final network."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -44,7 +29,8 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install semantica\n"
|
||||
"# Installation & Setup\n",
|
||||
"!pip install semantica mcp"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -53,38 +39,29 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, StreamIngestor, FeedIngestor\n",
|
||||
"from semantica.parse import DocumentParser, StructuredDataParser\n",
|
||||
"from semantica.kg import GraphBuilder, EntityResolver, ProvenanceTracker\n",
|
||||
"import tempfile\n",
|
||||
"import os\n",
|
||||
"import json\n",
|
||||
"import sqlite3\n",
|
||||
"import tempfile\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"file_ingestor = FileIngestor()\n",
|
||||
"web_ingestor = WebIngestor()\n",
|
||||
"db_ingestor = DBIngestor()\n",
|
||||
"stream_ingestor = StreamIngestor()\n",
|
||||
"feed_ingestor = FeedIngestor()\n",
|
||||
"# Import Semantica Modules\n",
|
||||
"from semantica.ingest import DBIngestor, WebIngestor, FileIngestor, MCPIngestor\n",
|
||||
"from semantica.kg import GraphBuilder\n",
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"\n",
|
||||
"temp_dir = tempfile.mkdtemp()\n",
|
||||
"\n",
|
||||
"file1 = os.path.join(temp_dir, \"source1.txt\")\n",
|
||||
"with open(file1, 'w') as f:\n",
|
||||
" f.write(\"Apple Inc. is a technology company. Tim Cook is the CEO.\")\n",
|
||||
"\n",
|
||||
"file_objects = file_ingestor.ingest_file(file1, read_content=True)\n",
|
||||
"\n",
|
||||
"print(f\"Ingested {len([file_objects]) if file_objects else 0} files\")\n",
|
||||
"print(f\"Multi-source ingestion initialized\")\n"
|
||||
"# Create a temporary workspace\n",
|
||||
"WORKSPACE_DIR = tempfile.mkdtemp()\n",
|
||||
"print(f\"Workspace created at: {WORKSPACE_DIR}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 2: Entity Resolution\n",
|
||||
"## Phase 1: Creating Data Sources\n",
|
||||
"\n",
|
||||
"Resolve entities across multiple sources.\n"
|
||||
"We generate files on disk to simulate the disparate enterprise systems."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -93,33 +70,53 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"entity_resolver = EntityResolver()\n",
|
||||
"# 1. SQLite Database (Financials)\n",
|
||||
"db_path = os.path.join(WORKSPACE_DIR, \"corporate.db\")\n",
|
||||
"conn = sqlite3.connect(db_path)\n",
|
||||
"conn.execute(\"CREATE TABLE financials (company_name TEXT, revenue REAL, employees INTEGER)\")\n",
|
||||
"conn.execute(\"INSERT INTO financials VALUES ('Nexus AI', 5500000.00, 45)\")\n",
|
||||
"conn.commit()\n",
|
||||
"conn.close()\n",
|
||||
"\n",
|
||||
"entities_from_source1 = [\n",
|
||||
" {\"id\": \"e1\", \"name\": \"Apple Inc.\", \"type\": \"Organization\", \"source\": \"file1\"},\n",
|
||||
" {\"id\": \"e2\", \"name\": \"Tim Cook\", \"type\": \"Person\", \"source\": \"file1\"}\n",
|
||||
"]\n",
|
||||
"# 2. Public Web (HTML)\n",
|
||||
"html_path = os.path.join(WORKSPACE_DIR, \"news.html\")\n",
|
||||
"with open(html_path, \"w\") as f:\n",
|
||||
" f.write(\"\"\"\n",
|
||||
" <html><body>\n",
|
||||
" <h1>Nexus AI Raises Series B</h1>\n",
|
||||
" <p>Nexus AI (San Francisco) valuated at $100M.</p>\n",
|
||||
" <p>CEO Jane Doe announces expansion.</p>\n",
|
||||
" </body></html>\n",
|
||||
" \"\"\")\n",
|
||||
"\n",
|
||||
"entities_from_source2 = [\n",
|
||||
" {\"id\": \"e3\", \"name\": \"Apple Incorporated\", \"type\": \"Organization\", \"source\": \"web\"},\n",
|
||||
" {\"id\": \"e4\", \"name\": \"Timothy Cook\", \"type\": \"Person\", \"source\": \"web\"}\n",
|
||||
"]\n",
|
||||
"# 3. Code Repository (Markdown)\n",
|
||||
"repo_path = os.path.join(WORKSPACE_DIR, \"README.md\")\n",
|
||||
"with open(repo_path, \"w\") as f:\n",
|
||||
" f.write(\"\"\"\n",
|
||||
" # Nexus AI Core\n",
|
||||
" Maintained by: engineering@nexus.ai\n",
|
||||
" Language: Python\n",
|
||||
" \"\"\")\n",
|
||||
"\n",
|
||||
"all_entities = entities_from_source1 + entities_from_source2\n",
|
||||
"# 4. Market Data API (JSON)\n",
|
||||
"api_path = os.path.join(WORKSPACE_DIR, \"market.json\")\n",
|
||||
"with open(api_path, \"w\") as f:\n",
|
||||
" json.dump({\n",
|
||||
" \"ticker\": \"NXAI\", \n",
|
||||
" \"price\": 124.50, \n",
|
||||
" \"employees\": 50 # Conflict with DB (45)\n",
|
||||
" }, f)\n",
|
||||
"\n",
|
||||
"resolved_entities = entity_resolver.resolve(all_entities)\n",
|
||||
"\n",
|
||||
"print(f\"Original entities: {len(all_entities)}\")\n",
|
||||
"print(f\"Resolved entities: {len(resolved_entities)}\")\n"
|
||||
"print(\"Data sources created successfully.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 3: Conflict Detection\n",
|
||||
"## Phase 2: Ingestion & Extraction\n",
|
||||
"\n",
|
||||
"Detect conflicts between sources.\n"
|
||||
"We use Semantica's ingestors to load data. In a real pipeline, we would attach an extractor (like an LLM) to parse the raw content into entities. Here, we simulate the extraction output for clarity."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -128,22 +125,140 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"conflict_detector = ConflictDetector()\n",
|
||||
"# Simulate extracted entities from our sources\n",
|
||||
"\n",
|
||||
"conflicts = conflict_detector.detect_value_conflicts(all_entities, \"name\")\n",
|
||||
"source_db = {\n",
|
||||
" \"name\": \"Corporate Database\",\n",
|
||||
" \"type\": \"structured\",\n",
|
||||
" \"entities\": [\n",
|
||||
" {\n",
|
||||
" \"name\": \"Nexus AI\",\n",
|
||||
" \"type\": \"Organization\",\n",
|
||||
" \"properties\": {\"revenue\": 5500000.00, \"employees\": 45},\n",
|
||||
" \"source\": \"corporate_db\"\n",
|
||||
" }\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"print(f\"Detected {len(conflicts)} conflicts\")\n",
|
||||
"for conflict in conflicts[:3]:\n",
|
||||
" print(f\" Conflict: {conflict.entity_id} - {conflict.conflict_type}\")\n"
|
||||
"source_web = {\n",
|
||||
" \"name\": \"Web News\",\n",
|
||||
" \"type\": \"unstructured\",\n",
|
||||
" \"entities\": [\n",
|
||||
" {\n",
|
||||
" \"name\": \"Nexus AI\",\n",
|
||||
" \"type\": \"Organization\",\n",
|
||||
" \"properties\": {\"valuation\": \"$100M\", \"location\": \"San Francisco\"},\n",
|
||||
" \"source\": \"public_web\"\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"name\": \"Jane Doe\",\n",
|
||||
" \"type\": \"Person\",\n",
|
||||
" \"properties\": {\"role\": \"CEO\"},\n",
|
||||
" \"source\": \"public_web\"\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"relationships\": [\n",
|
||||
" {\"source\": \"Jane Doe\", \"target\": \"Nexus AI\", \"type\": \"is_ceo_of\"}\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"source_api = {\n",
|
||||
" \"name\": \"Market API\",\n",
|
||||
" \"type\": \"structured\",\n",
|
||||
" \"entities\": [\n",
|
||||
" {\n",
|
||||
" \"name\": \"Nexus AI\",\n",
|
||||
" \"type\": \"Organization\",\n",
|
||||
" \"properties\": {\"ticker\": \"NXAI\", \"employees\": 50}, # Note conflict: 50 vs 45\n",
|
||||
" \"source\": \"market_api\"\n",
|
||||
" }\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"source_repo = {\n",
|
||||
" \"name\": \"GitHub Repo\",\n",
|
||||
" \"type\": \"semi-structured\",\n",
|
||||
" \"entities\": [\n",
|
||||
" {\n",
|
||||
" \"name\": \"Nexus AI Core\",\n",
|
||||
" \"type\": \"Software\",\n",
|
||||
" \"properties\": {\"language\": \"Python\"},\n",
|
||||
" \"source\": \"github\"\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"relationships\": [\n",
|
||||
" {\"source\": \"Nexus AI Core\", \"target\": \"Nexus AI\", \"type\": \"owned_by\"}\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# 5. Web Search MCP\n",
|
||||
"# We use Semantica's MCPIngestor to connect to a Web Search MCP server.\n",
|
||||
"# This allows us to fetch live competitor data (e.g., from Brave Search).\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" # Initialize MCP Ingestor\n",
|
||||
" mcp = MCPIngestor()\n",
|
||||
" \n",
|
||||
" # Attempt to connect to a local MCP server (e.g., running on port 8000)\n",
|
||||
" # Example: `fastmcp run search_server.py`\n",
|
||||
" mcp.connect(\"web_search\", url=\"http://localhost:8000/sse\")\n",
|
||||
" \n",
|
||||
" print(\"Connected to MCP Server. Ingesting live data...\")\n",
|
||||
" \n",
|
||||
" # Ingest data using the search tool\n",
|
||||
" mcp_data = mcp.ingest_tool_output(\n",
|
||||
" \"web_search\", \n",
|
||||
" \"search\", \n",
|
||||
" {\"query\": \"Nexus AI competitors valuation\"}\n",
|
||||
" )\n",
|
||||
" \n",
|
||||
" # Process the output (assuming the tool returns structured entities)\n",
|
||||
" # In a real app, you might need an LLM to extract entities from search results.\n",
|
||||
" # Here we assume the MCP server returns ready-to-use entities.\n",
|
||||
" source_mcp = {\n",
|
||||
" \"name\": \"Web Search MCP\",\n",
|
||||
" \"type\": \"agent-tool\",\n",
|
||||
" \"entities\": mcp_data.get(\"entities\", []),\n",
|
||||
" \"source\": \"mcp_search\"\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
"except Exception as e:\n",
|
||||
" print(f\"MCP Server connection failed ({e}). Using simulated data.\")\n",
|
||||
" print(\"To enable live data, ensure an MCP server is running at http://localhost:8000/sse\")\n",
|
||||
" \n",
|
||||
" # Fallback to simulated data\n",
|
||||
" source_mcp = {\n",
|
||||
" \"name\": \"Web Search MCP\",\n",
|
||||
" \"type\": \"agent-tool\",\n",
|
||||
" \"entities\": [\n",
|
||||
" {\n",
|
||||
" \"name\": \"Nexus AI\",\n",
|
||||
" \"type\": \"Organization\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"competitors\": [\"Cyberdyne Systems\", \"Massive Dynamic\"],\n",
|
||||
" \"valuation\": \"$120M\" # Note conflict: $120M (newer) vs $100M (older web)\n",
|
||||
" },\n",
|
||||
" \"source\": \"mcp_search_agent\"\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"relationships\": [\n",
|
||||
" {\"source\": \"Nexus AI\", \"target\": \"Cyberdyne Systems\", \"type\": \"competes_with\"}\n",
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"all_sources = [source_db, source_web, source_api, source_repo, source_mcp]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 4: Provenance Tracking\n",
|
||||
"## Phase 3: Graph Construction & Resolution\n",
|
||||
"\n",
|
||||
"Track data provenance across sources.\n"
|
||||
"We use `GraphBuilder` to:\n",
|
||||
"1. **Merge Entities**: Combine the 4 \"Nexus AI\" records into one canonical node.\n",
|
||||
"2. **Resolve Conflicts**: Handle discrepancies (Employee count: 45 vs 50; Valuation: $100M vs $120M).\n",
|
||||
"3. **Build Graph**: Link related entities (CEO, Software, Competitors)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -152,28 +267,34 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"provenance_tracker = ProvenanceTracker()\n",
|
||||
"# Initialize GraphBuilder with resolution enabled\n",
|
||||
"builder = GraphBuilder(\n",
|
||||
" merge_entities=True,\n",
|
||||
" resolve_conflicts=True,\n",
|
||||
" entity_resolution_strategy=\"fuzzy\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"for entity in all_entities:\n",
|
||||
" provenance_tracker.track_entity(entity.get(\"id\"), entity.get(\"source\"), entity)\n",
|
||||
"# Build the graph\n",
|
||||
"print(\"Building Knowledge Graph...\")\n",
|
||||
"kg = builder.build(sources=all_sources)\n",
|
||||
"\n",
|
||||
"relationships = [\n",
|
||||
" {\"source\": \"e2\", \"target\": \"e1\", \"type\": \"CEO_of\", \"source\": \"file1\"}\n",
|
||||
"]\n",
|
||||
"print(f\"Graph built with {len(kg['nodes'])} nodes and {len(kg['edges'])} edges.\")\n",
|
||||
"\n",
|
||||
"for rel in relationships:\n",
|
||||
" provenance_tracker.track_relationship(rel.get(\"source\"), rel.get(\"target\"), rel.get(\"source\"), rel)\n",
|
||||
"\n",
|
||||
"print(f\"Tracked provenance for {len(all_entities)} entities and {len(relationships)} relationships\")\n"
|
||||
"# Verify conflict resolution results\n",
|
||||
"nexus_node = next(n for n in kg['nodes'] if n['name'] == \"Nexus AI\")\n",
|
||||
"print(\"\\nResolved Properties for Nexus AI:\")\n",
|
||||
"print(f\" Employees: {nexus_node['properties'].get('employees')} (Resolved from DB/API)\")\n",
|
||||
"print(f\" Valuation: {nexus_node['properties'].get('valuation')} (Resolved from Web/MCP)\")\n",
|
||||
"print(f\" Competitors: {nexus_node['properties'].get('competitors')}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 5: Build Unified Knowledge Graph\n",
|
||||
"## Phase 4: Visualization\n",
|
||||
"\n",
|
||||
"Build a unified knowledge graph from integrated sources.\n"
|
||||
"We use `KGVisualizer` to render the interactive graph."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -182,37 +303,48 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"builder = GraphBuilder()\n",
|
||||
"print(\"Visualizing Graph...\")\n",
|
||||
"visualizer = KGVisualizer(layout=\"force\", color_scheme=\"vibrant\")\n",
|
||||
"\n",
|
||||
"unified_kg = builder.build(resolved_entities, relationships)\n",
|
||||
"\n",
|
||||
"print(f\"Built unified knowledge graph\")\n",
|
||||
"print(f\" Entities: {len(unified_kg.get('entities', []))}\")\n",
|
||||
"print(f\" Relationships: {len(unified_kg.get('relationships', []))}\")\n",
|
||||
"print(f\" Sources integrated: {len(set(e.get('source', '') for e in resolved_entities))}\")\n"
|
||||
"# Render the network\n",
|
||||
"# This supports interactive output in Jupyter\n",
|
||||
"fig = visualizer.visualize_network(kg, output=\"interactive\")\n",
|
||||
"fig.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"You've learned advanced multi-source data integration:\n",
|
||||
"\n",
|
||||
"- **Multiple Ingestion Types**: FileIngestor, WebIngestor, DBIngestor, StreamIngestor, FeedIngestor\n",
|
||||
"- **EntityResolver**: Resolve entities across sources\n",
|
||||
"- **ConflictDetector**: Detect conflicts between sources\n",
|
||||
"- **ProvenanceTracker**: Track data provenance\n",
|
||||
"- **Unified Knowledge Graph**: Build integrated graph from multiple sources\n"
|
||||
"In this notebook, we used `semantica`'s high-level modules to:\n",
|
||||
"1. Ingest data from 5 different sources (including a simulated **MCP Search Agent**).\n",
|
||||
"2. Automatically resolve identity to create a single \"Nexus AI\" node.\n",
|
||||
"3. Resolve complex data conflicts (Valuation, Employee Count).\n",
|
||||
"4. Visualize the unified Knowledge Graph with competitor relationships.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
Reference in New Issue
Block a user