Nexus AI (San Francisco) valuated at $100M.
\n", + "CEO Jane Doe announces expansion.
\n", + " \n", + " \"\"\")\n", "\n", - "entities_from_source2 = [\n", - " {\"id\": \"e3\", \"name\": \"Apple Incorporated\", \"type\": \"Organization\", \"source\": \"web\"},\n", - " {\"id\": \"e4\", \"name\": \"Timothy Cook\", \"type\": \"Person\", \"source\": \"web\"}\n", - "]\n", + "# 3. Code Repository (Markdown)\n", + "repo_path = os.path.join(WORKSPACE_DIR, \"README.md\")\n", + "with open(repo_path, \"w\") as f:\n", + " f.write(\"\"\"\n", + " # Nexus AI Core\n", + " Maintained by: engineering@nexus.ai\n", + " Language: Python\n", + " \"\"\")\n", "\n", - "all_entities = entities_from_source1 + entities_from_source2\n", + "# 4. Market Data API (JSON)\n", + "api_path = os.path.join(WORKSPACE_DIR, \"market.json\")\n", + "with open(api_path, \"w\") as f:\n", + " json.dump({\n", + " \"ticker\": \"NXAI\", \n", + " \"price\": 124.50, \n", + " \"employees\": 50 # Conflict with DB (45)\n", + " }, f)\n", "\n", - "resolved_entities = entity_resolver.resolve(all_entities)\n", - "\n", - "print(f\"Original entities: {len(all_entities)}\")\n", - "print(f\"Resolved entities: {len(resolved_entities)}\")\n" + "print(\"Data sources created successfully.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 3: Conflict Detection\n", + "## Phase 2: Ingestion & Extraction\n", "\n", - "Detect conflicts between sources.\n" + "We use Semantica's ingestors to load data. In a real pipeline, we would attach an extractor (like an LLM) to parse the raw content into entities. Here, we simulate the extraction output for clarity." ] }, { @@ -128,22 +125,140 @@ "metadata": {}, "outputs": [], "source": [ - "conflict_detector = ConflictDetector()\n", + "# Simulate extracted entities from our sources\n", "\n", - "conflicts = conflict_detector.detect_value_conflicts(all_entities, \"name\")\n", + "source_db = {\n", + " \"name\": \"Corporate Database\",\n", + " \"type\": \"structured\",\n", + " \"entities\": [\n", + " {\n", + " \"name\": \"Nexus AI\",\n", + " \"type\": \"Organization\",\n", + " \"properties\": {\"revenue\": 5500000.00, \"employees\": 45},\n", + " \"source\": \"corporate_db\"\n", + " }\n", + " ]\n", + "}\n", "\n", - "print(f\"Detected {len(conflicts)} conflicts\")\n", - "for conflict in conflicts[:3]:\n", - " print(f\" Conflict: {conflict.entity_id} - {conflict.conflict_type}\")\n" + "source_web = {\n", + " \"name\": \"Web News\",\n", + " \"type\": \"unstructured\",\n", + " \"entities\": [\n", + " {\n", + " \"name\": \"Nexus AI\",\n", + " \"type\": \"Organization\",\n", + " \"properties\": {\"valuation\": \"$100M\", \"location\": \"San Francisco\"},\n", + " \"source\": \"public_web\"\n", + " },\n", + " {\n", + " \"name\": \"Jane Doe\",\n", + " \"type\": \"Person\",\n", + " \"properties\": {\"role\": \"CEO\"},\n", + " \"source\": \"public_web\"\n", + " }\n", + " ],\n", + " \"relationships\": [\n", + " {\"source\": \"Jane Doe\", \"target\": \"Nexus AI\", \"type\": \"is_ceo_of\"}\n", + " ]\n", + "}\n", + "\n", + "source_api = {\n", + " \"name\": \"Market API\",\n", + " \"type\": \"structured\",\n", + " \"entities\": [\n", + " {\n", + " \"name\": \"Nexus AI\",\n", + " \"type\": \"Organization\",\n", + " \"properties\": {\"ticker\": \"NXAI\", \"employees\": 50}, # Note conflict: 50 vs 45\n", + " \"source\": \"market_api\"\n", + " }\n", + " ]\n", + "}\n", + "\n", + "source_repo = {\n", + " \"name\": \"GitHub Repo\",\n", + " \"type\": \"semi-structured\",\n", + " \"entities\": [\n", + " {\n", + " \"name\": \"Nexus AI Core\",\n", + " \"type\": \"Software\",\n", + " \"properties\": {\"language\": \"Python\"},\n", + " \"source\": \"github\"\n", + " }\n", + " ],\n", + " \"relationships\": [\n", + " {\"source\": \"Nexus AI Core\", \"target\": \"Nexus AI\", \"type\": \"owned_by\"}\n", + " ]\n", + "}\n", + "\n", + "# 5. Web Search MCP\n", + "# We use Semantica's MCPIngestor to connect to a Web Search MCP server.\n", + "# This allows us to fetch live competitor data (e.g., from Brave Search).\n", + "\n", + "try:\n", + " # Initialize MCP Ingestor\n", + " mcp = MCPIngestor()\n", + " \n", + " # Attempt to connect to a local MCP server (e.g., running on port 8000)\n", + " # Example: `fastmcp run search_server.py`\n", + " mcp.connect(\"web_search\", url=\"http://localhost:8000/sse\")\n", + " \n", + " print(\"Connected to MCP Server. Ingesting live data...\")\n", + " \n", + " # Ingest data using the search tool\n", + " mcp_data = mcp.ingest_tool_output(\n", + " \"web_search\", \n", + " \"search\", \n", + " {\"query\": \"Nexus AI competitors valuation\"}\n", + " )\n", + " \n", + " # Process the output (assuming the tool returns structured entities)\n", + " # In a real app, you might need an LLM to extract entities from search results.\n", + " # Here we assume the MCP server returns ready-to-use entities.\n", + " source_mcp = {\n", + " \"name\": \"Web Search MCP\",\n", + " \"type\": \"agent-tool\",\n", + " \"entities\": mcp_data.get(\"entities\", []),\n", + " \"source\": \"mcp_search\"\n", + " }\n", + "\n", + "except Exception as e:\n", + " print(f\"MCP Server connection failed ({e}). Using simulated data.\")\n", + " print(\"To enable live data, ensure an MCP server is running at http://localhost:8000/sse\")\n", + " \n", + " # Fallback to simulated data\n", + " source_mcp = {\n", + " \"name\": \"Web Search MCP\",\n", + " \"type\": \"agent-tool\",\n", + " \"entities\": [\n", + " {\n", + " \"name\": \"Nexus AI\",\n", + " \"type\": \"Organization\",\n", + " \"properties\": {\n", + " \"competitors\": [\"Cyberdyne Systems\", \"Massive Dynamic\"],\n", + " \"valuation\": \"$120M\" # Note conflict: $120M (newer) vs $100M (older web)\n", + " },\n", + " \"source\": \"mcp_search_agent\"\n", + " }\n", + " ],\n", + " \"relationships\": [\n", + " {\"source\": \"Nexus AI\", \"target\": \"Cyberdyne Systems\", \"type\": \"competes_with\"}\n", + " ]\n", + "}\n", + "\n", + "all_sources = [source_db, source_web, source_api, source_repo, source_mcp]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 4: Provenance Tracking\n", + "## Phase 3: Graph Construction & Resolution\n", "\n", - "Track data provenance across sources.\n" + "We use `GraphBuilder` to:\n", + "1. **Merge Entities**: Combine the 4 \"Nexus AI\" records into one canonical node.\n", + "2. **Resolve Conflicts**: Handle discrepancies (Employee count: 45 vs 50; Valuation: $100M vs $120M).\n", + "3. **Build Graph**: Link related entities (CEO, Software, Competitors)." ] }, { @@ -152,28 +267,34 @@ "metadata": {}, "outputs": [], "source": [ - "provenance_tracker = ProvenanceTracker()\n", + "# Initialize GraphBuilder with resolution enabled\n", + "builder = GraphBuilder(\n", + " merge_entities=True,\n", + " resolve_conflicts=True,\n", + " entity_resolution_strategy=\"fuzzy\"\n", + ")\n", "\n", - "for entity in all_entities:\n", - " provenance_tracker.track_entity(entity.get(\"id\"), entity.get(\"source\"), entity)\n", + "# Build the graph\n", + "print(\"Building Knowledge Graph...\")\n", + "kg = builder.build(sources=all_sources)\n", "\n", - "relationships = [\n", - " {\"source\": \"e2\", \"target\": \"e1\", \"type\": \"CEO_of\", \"source\": \"file1\"}\n", - "]\n", + "print(f\"Graph built with {len(kg['nodes'])} nodes and {len(kg['edges'])} edges.\")\n", "\n", - "for rel in relationships:\n", - " provenance_tracker.track_relationship(rel.get(\"source\"), rel.get(\"target\"), rel.get(\"source\"), rel)\n", - "\n", - "print(f\"Tracked provenance for {len(all_entities)} entities and {len(relationships)} relationships\")\n" + "# Verify conflict resolution results\n", + "nexus_node = next(n for n in kg['nodes'] if n['name'] == \"Nexus AI\")\n", + "print(\"\\nResolved Properties for Nexus AI:\")\n", + "print(f\" Employees: {nexus_node['properties'].get('employees')} (Resolved from DB/API)\")\n", + "print(f\" Valuation: {nexus_node['properties'].get('valuation')} (Resolved from Web/MCP)\")\n", + "print(f\" Competitors: {nexus_node['properties'].get('competitors')}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 5: Build Unified Knowledge Graph\n", + "## Phase 4: Visualization\n", "\n", - "Build a unified knowledge graph from integrated sources.\n" + "We use `KGVisualizer` to render the interactive graph." ] }, { @@ -182,37 +303,48 @@ "metadata": {}, "outputs": [], "source": [ - "builder = GraphBuilder()\n", + "print(\"Visualizing Graph...\")\n", + "visualizer = KGVisualizer(layout=\"force\", color_scheme=\"vibrant\")\n", "\n", - "unified_kg = builder.build(resolved_entities, relationships)\n", - "\n", - "print(f\"Built unified knowledge graph\")\n", - "print(f\" Entities: {len(unified_kg.get('entities', []))}\")\n", - "print(f\" Relationships: {len(unified_kg.get('relationships', []))}\")\n", - "print(f\" Sources integrated: {len(set(e.get('source', '') for e in resolved_entities))}\")\n" + "# Render the network\n", + "# This supports interactive output in Jupyter\n", + "fig = visualizer.visualize_network(kg, output=\"interactive\")\n", + "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Summary\n", + "## Conclusion\n", "\n", - "You've learned advanced multi-source data integration:\n", - "\n", - "- **Multiple Ingestion Types**: FileIngestor, WebIngestor, DBIngestor, StreamIngestor, FeedIngestor\n", - "- **EntityResolver**: Resolve entities across sources\n", - "- **ConflictDetector**: Detect conflicts between sources\n", - "- **ProvenanceTracker**: Track data provenance\n", - "- **Unified Knowledge Graph**: Build integrated graph from multiple sources\n" + "In this notebook, we used `semantica`'s high-level modules to:\n", + "1. Ingest data from 5 different sources (including a simulated **MCP Search Agent**).\n", + "2. Automatically resolve identity to create a single \"Nexus AI\" node.\n", + "3. Resolve complex data conflicts (Valuation, Employee Count).\n", + "4. Visualize the unified Knowledge Graph with competitor relationships.\n" ] } ], "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, "language_info": { - "name": "python" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" } }, "nbformat": 4, - "nbformat_minor": 2 -} + "nbformat_minor": 4 +} \ No newline at end of file