semantica/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)\n",
    "\n",
    "# Multi-Source Data Integration\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates advanced multi-source data integration using multiple ingestion types, entity resolution, conflict detection, and provenance tracking.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/ingest/)\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest data from multiple sources (files, web, databases, streams, feeds)\n",
    "- Resolve entities across sources using EntityResolver\n",
    "- Detect conflicts using ConflictDetector\n",
    "- Track provenance using ProvenanceTracker\n",
    "- Integrate data into a unified knowledge graph\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Workflow: Multi-Source Ingestion → Entity Resolution → Conflict Detection → Provenance Tracking → Unified KG\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, StreamIngestor, FeedIngestor\n",
    "from semantica.parse import DocumentParser, StructuredDataParser\n",
    "from semantica.kg import GraphBuilder, EntityResolver, ProvenanceTracker\n",
    "from semantica.conflicts import ConflictDetector\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "stream_ingestor = StreamIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "file1 = os.path.join(temp_dir, \"source1.txt\")\n",
    "with open(file1, 'w') as f:\n",
    "    f.write(\"Apple Inc. is a technology company. Tim Cook is the CEO.\")\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(file1, read_content=True)\n",
    "\n",
    "print(f\"Ingested {len([file_objects]) if file_objects else 0} files\")\n",
    "print(f\"Multi-source ingestion initialized\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Entity Resolution\n",
    "\n",
    "Resolve entities across multiple sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "entity_resolver = EntityResolver()\n",
    "\n",
    "entities_from_source1 = [\n",
    "    {\"id\": \"e1\", \"name\": \"Apple Inc.\", \"type\": \"Organization\", \"source\": \"file1\"},\n",
    "    {\"id\": \"e2\", \"name\": \"Tim Cook\", \"type\": \"Person\", \"source\": \"file1\"}\n",
    "]\n",
    "\n",
    "entities_from_source2 = [\n",
    "    {\"id\": \"e3\", \"name\": \"Apple Incorporated\", \"type\": \"Organization\", \"source\": \"web\"},\n",
    "    {\"id\": \"e4\", \"name\": \"Timothy Cook\", \"type\": \"Person\", \"source\": \"web\"}\n",
    "]\n",
    "\n",
    "all_entities = entities_from_source1 + entities_from_source2\n",
    "\n",
    "resolved_entities = entity_resolver.resolve(all_entities)\n",
    "\n",
    "print(f\"Original entities: {len(all_entities)}\")\n",
    "print(f\"Resolved entities: {len(resolved_entities)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Conflict Detection\n",
    "\n",
    "Detect conflicts between sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conflict_detector = ConflictDetector()\n",
    "\n",
    "conflicts = conflict_detector.detect_value_conflicts(all_entities, \"name\")\n",
    "\n",
    "print(f\"Detected {len(conflicts)} conflicts\")\n",
    "for conflict in conflicts[:3]:\n",
    "    print(f\"  Conflict: {conflict.entity_id} - {conflict.conflict_type}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Provenance Tracking\n",
    "\n",
    "Track data provenance across sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "provenance_tracker = ProvenanceTracker()\n",
    "\n",
    "for entity in all_entities:\n",
    "    provenance_tracker.track_entity(entity.get(\"id\"), entity.get(\"source\"), entity)\n",
    "\n",
    "relationships = [\n",
    "    {\"source\": \"e2\", \"target\": \"e1\", \"type\": \"CEO_of\", \"source\": \"file1\"}\n",
    "]\n",
    "\n",
    "for rel in relationships:\n",
    "    provenance_tracker.track_relationship(rel.get(\"source\"), rel.get(\"target\"), rel.get(\"source\"), rel)\n",
    "\n",
    "print(f\"Tracked provenance for {len(all_entities)} entities and {len(relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Build Unified Knowledge Graph\n",
    "\n",
    "Build a unified knowledge graph from integrated sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "\n",
    "unified_kg = builder.build(resolved_entities, relationships)\n",
    "\n",
    "print(f\"Built unified knowledge graph\")\n",
    "print(f\"  Entities: {len(unified_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(unified_kg.get('relationships', []))}\")\n",
    "print(f\"  Sources integrated: {len(set(e.get('source', '') for e in resolved_entities))}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "You've learned advanced multi-source data integration:\n",
    "\n",
    "- **Multiple Ingestion Types**: FileIngestor, WebIngestor, DBIngestor, StreamIngestor, FeedIngestor\n",
    "- **EntityResolver**: Resolve entities across sources\n",
    "- **ConflictDetector**: Detect conflicts between sources\n",
    "- **ProvenanceTracker**: Track data provenance\n",
    "- **Unified Knowledge Graph**: Build integrated graph from multiple sources\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}