{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
    "\n",
    "# Deduplication in Semantica\n",
    "\n",
    "Welcome to the **Deduplication** walkthrough! In any Knowledge Graph, data often comes from multiple sources, leading to duplicate entities (e.g., \"Apple Inc.\" vs. \"Apple Inc\"). \n",
    "\n",
    "Semantica provides a robust **Deduplication Module** to help you:\n",
    "1. **Calculate Similarity**: Compare entities using strings, properties, and embeddings.\n",
    "2. **Detect Duplicates**: Find pairs or groups of entities that represent the same real-world object.\n",
    "3. **Cluster Entities**: Group similar entities together.\n",
    "4. **Merge Entities**: Combine duplicates into a single, canonical entity while resolving conflicts.\n",
    "\n",
    "This notebook will guide you through each step with clear examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install Semantica\n",
    "!pip install -q semantica"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparing Sample Data\n",
    "\n",
    "Let's create a dataset with some intentional duplicates. We'll simulate data coming from different sources (e.g., a CRM and a public database).\n",
    "\n",
    "**Our Entities:**\n",
    "- **Apple**: Variations like \"Apple Inc.\", \"Apple Inc\", and just \"Apple\".\n",
    "- **Microsoft**: Variations like \"Microsoft Corp\" and \"Microsoft\".\n",
    "- **Google**: A unique entity for control."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "entities = [\n",
    "    # Apple Variations\n",
    "    {\n",
    "        \"id\": \"e1\",\n",
    "        \"name\": \"Apple Inc.\",\n",
    "        \"type\": \"Company\",\n",
    "        \"properties\": {\"industry\": \"Technology\", \"hq\": \"Cupertino\", \"founded\": 1976},\n",
    "        \"relationships\": [{\"predicate\": \"founded_by\", \"object\": \"Steve Jobs\"}]\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"e2\",\n",
    "        \"name\": \"Apple Inc\",\n",
    "        \"type\": \"Company\",\n",
    "        \"properties\": {\"industry\": \"Tech\", \"hq\": \"Cupertino, CA\"}, # Slightly different properties\n",
    "        \"relationships\": []\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"e3\",\n",
    "        \"name\": \"Apple\",\n",
    "        \"type\": \"Company\",\n",
    "        \"properties\": {\"industry\": \"Consumer Electronics\"}, \n",
    "        \"relationships\": [{\"predicate\": \"ceo\", \"object\": \"Tim Cook\"}]\n",
    "    },\n",
    "    \n",
    "    # Microsoft Variations\n",
    "    {\n",
    "        \"id\": \"e4\",\n",
    "        \"name\": \"Microsoft Corp\",\n",
    "        \"type\": \"Company\",\n",
    "        \"properties\": {\"industry\": \"Software\", \"hq\": \"Redmond\"}\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"e5\",\n",
    "        \"name\": \"Microsoft\",\n",
    "        \"type\": \"Company\",\n",
    "        \"properties\": {\"industry\": \"Tech\", \"hq\": \"Redmond, WA\"}\n",
    "    },\n",
    "    \n",
    "    # Unique Entity\n",
    "    {\n",
    "        \"id\": \"e6\",\n",
    "        \"name\": \"Google LLC\",\n",
    "        \"type\": \"Company\",\n",
    "        \"properties\": {\"industry\": \"Internet\"}\n",
    "    }\n",
    "]\n",
    "\n",
    "print(f\"Created {len(entities)} sample entities.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Similarity Calculation\n",
    "\n",
    "The `SimilarityCalculator` is the core engine. It compares two entities and returns a score between 0 and 1. It looks at:\n",
    "- **String Similarity**: Names and text fields.\n",
    "- **Property Similarity**: Overlap in key-value pairs.\n",
    "- **Relationship Similarity**: Connections to other entities.\n",
    "- **Embeddings**: Semantic vector similarity (if available).\n",
    "\n",
    "You can customize the weights for each factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.deduplication import SimilarityCalculator, SimilarityResult\n",
    "# Initialize calculator with custom weights\n",
    "calculator = SimilarityCalculator(\n",
    "    string_weight=0.5,      # High importance on name\n",
    "    property_weight=0.3,    # Medium importance on properties\n",
    "    relationship_weight=0.2 # Lower importance on relationships\n",
    ")\n",
    "\n",
    "# Compare \"Apple Inc.\" (e1) vs \"Apple Inc\" (e2)\n",
    "score_e1_e2 = calculator.calculate_similarity(entities[0], entities[1])\n",
    "\n",
    "print(f\"Similarity between '{entities[0]['name']}' and '{entities[1]['name']}':\")\n",
    "print(f\"  Total Score: {score_e1_e2.score:.4f}\")\n",
    "print(f\"  Breakdown:   {score_e1_e2.components}\")\n",
    "\n",
    "# Compare \"Apple Inc.\" (e1) vs \"Microsoft\" (e5)\n",
    "score_e1_e5 = calculator.calculate_similarity(entities[0], entities[4])\n",
    "\n",
    "print(f\"\\nSimilarity between '{entities[0]['name']}' and '{entities[4]['name']}':\")\n",
    "print(f\"  Total Score: {score_e1_e5.score:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Duplicate Detection\n",
    "\n",
    "The `DuplicateDetector` uses the similarity calculator to scan your dataset for duplicates. It can find:\n",
    "- **Pairs**: Simple A matches B.\n",
    "- **Groups**: A matches B, and B matches C.\n",
    "\n",
    "It uses a `similarity_threshold` to decide what counts as a match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import specific classes for Duplicate Detection\n",
    "from semantica.deduplication import DuplicateDetector, DuplicateCandidate, DuplicateGroup\n",
    "from semantica.deduplication import DeduplicationConfig\n",
    "\n",
    "detector = DuplicateDetector(\n",
    "    similarity_threshold=0.7, \n",
    "    confidence_threshold=0.6  \n",
    ")\n",
    "\n",
    "# Detect pairs\n",
    "candidates = detector.detect_duplicates(entities)\n",
    "\n",
    "print(f\"Found {len(candidates)} duplicate pairs:\")\n",
    "for c in candidates:\n",
    "    print(f\"  - {c.entity1['name']} <==> {c.entity2['name']} (Score: {c.similarity_score:.2f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Incremental Detection\n",
    "If you have an existing database and ingest new data, you don't want to re-compare everything. Use `incremental_detect`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "existing_db = entities[:3] # The Apple entities\n",
    "new_data = [entities[4]]   # Microsoft\n",
    "# Check if new data matches anything in existing DB\n",
    "inc_candidates = detector.incremental_detect(new_data, existing_db)\n",
    "\n",
    "print(f\"New matches found: {len(inc_candidates)}\")\n",
    "# Expected: 0, because Microsoft is not Apple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clustering\n",
    "\n",
    "Sometimes pairs aren't enough. `ClusterBuilder` groups related entities into clusters. This is useful for understanding the full scope of a duplicated entity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import specific classes for Clustering\n",
    "from semantica.deduplication import ClusterBuilder, Cluster, ClusterResult\n",
    "cluster_builder = ClusterBuilder(threshold=0.7)\n",
    "result = cluster_builder.build_clusters(entities)\n",
    "\n",
    "print(f\"Found {len(result.clusters)} clusters:\")\n",
    "for i, cluster in enumerate(result.clusters):\n",
    "    names = [e['name'] for e in cluster.entities]\n",
    "    print(f\"  Cluster {i+1}: {names}\")\n",
    "\n",
    "cluster_builder = ClusterBuilder(threshold=0.7)\n",
    "result = cluster_builder.build_clusters(entities)\n",
    "\n",
    "print(f\"Found {len(result.clusters)} clusters:\")\n",
    "for i, cluster in enumerate(result.clusters):\n",
    "    names = [e['name'] for e in cluster.entities]\n",
    "    print(f\"  Cluster {i+1}: {names}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Entity Merging\n",
    "\n",
    "Once duplicates are found, `EntityMerger` combines them. You need to choose a **Merge Strategy**:\n",
    "\n",
    "- `KEEP_FIRST` / `KEEP_LAST`: Based on order.\n",
    "- `KEEP_MOST_COMPLETE`: Keeps the entity with the most data (properties + relationships).\n",
    "- `KEEP_HIGHEST_CONFIDENCE`: Uses internal confidence scores.\n",
    "- `MERGE_ALL`: Combines everything (arrays are concatenated, conflicts resolved by voting)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import specific classes for Entity Merging\n",
    "from semantica.deduplication import EntityMerger, MergeStrategy, MergeStrategyManager, MergeOperation, MergeResult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merger = EntityMerger()\n",
    "\n",
    "# We will use the 'KEEP_MOST_COMPLETE' strategy\n",
    "# This ensures we don't lose valuable information from richer entities\n",
    "merge_ops = merger.merge_duplicates(\n",
    "    entities, \n",
    "    strategy=MergeStrategy.KEEP_MOST_COMPLETE\n",
    ")\n",
    "\n",
    "print(f\"Performed {len(merge_ops)} merge operations.\")\n",
    "\n",
    "print(\"\\n--- Merged Results ---\")\n",
    "for op in merge_ops:\n",
    "    final_ent = op.merged_entity\n",
    "    original_count = len(op.source_entities)\n",
    "    print(f\"Merged {original_count} entities into: '{final_ent['name']}'\")\n",
    "    print(f\"  - Final Properties: {final_ent['properties']}\")\n",
    "    print(f\"  - Final Relationships: {len(final_ent.get('relationships', []))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Complete Workflow\n",
    "\n",
    "Let's wrap this up into a clean function that takes dirty data and returns clean data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deduplicate_dataset(raw_entities):\n",
    "    print(\"1. Detecting duplicates...\")\n",
    "    # Step 1: Detect\n",
    "    detector = DuplicateDetector(similarity_threshold=0.75)\n",
    "    # We can skip explicit detection calls if we just want to merge, \n",
    "    # as EntityMerger calls detection internally, but doing it manually allows inspection.\n",
    "    \n",
    "    print(\"2. Merging entities...\")\n",
    "    # Step 2: Merge\n",
    "    merger = EntityMerger()\n",
    "    ops = merger.merge_duplicates(raw_entities, strategy=MergeStrategy.KEEP_MOST_COMPLETE)\n",
    "    \n",
    "    # Let's collect all final IDs to see what remains\n",
    "    merged_entities = [op.merged_entity for op in ops]\n",
    "    \n",
    "    # Find entities that were NOT part of any merge (singletons)\n",
    "    merged_ids = set()\n",
    "    for op in ops:\n",
    "        for source in op.source_entities:\n",
    "            merged_ids.add(source['id'])\n",
    "            \n",
    "    singletons = [e for e in raw_entities if e['id'] not in merged_ids]\n",
    "    \n",
    "    final_dataset = merged_entities + singletons\n",
    "    return final_dataset\n",
    "\n",
    "# Run the workflow\n",
    "clean_data = deduplicate_dataset(entities)\n",
    "\n",
    "print(f\"\\nOriginal Size: {len(entities)}\")\n",
    "print(f\"Cleaned Size:  {len(clean_data)}\")\n",
    "print(\"\\nFinal Entity Names:\")\n",
    "for e in clean_data:\n",
    "    print(f\"  - {e['name']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "You've learned how to:\n",
    "1. **Import** the necessary Deduplication classes.\n",
    "2. **Calculate Similarity** between entities.\n",
    "3. **Detect Duplicates** using configurable thresholds.\n",
    "4. **Cluster** similar entities.\n",
    "5. **Merge** duplicates into a clean, canonical dataset.\n",
    "\n",
    "This module is essential for maintaining high-quality Knowledge Graphs, especially when ingesting data from multiple, potentially messy sources."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}