{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
    "\n",
    "# Deduplication Module\n",
    "\n",
    "## What is the Deduplication Module?\n",
    "\n",
    "The Deduplication Module is a comprehensive system for identifying and merging duplicate entities in knowledge graphs. It helps maintain data quality by detecting semantically similar entities, calculating similarity scores, and merging duplicates into canonical representations while preserving provenance and handling conflicts.\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/deduplication/)\n",
    "\n",
    "## Module Capabilities\n",
    "\n",
    "### Core Functionality\n",
    "\n",
    "1. **Similarity Calculation**\n",
    "   - Multiple algorithms: Exact matching, Levenshtein distance, Jaro-Winkler, Cosine similarity, Jaccard similarity\n",
    "   - Multi-factor aggregation: Combines string, property, relationship, and embedding similarity\n",
    "   - Configurable weights for different similarity components\n",
    "   - Batch processing for efficient pairwise comparisons\n",
    "\n",
    "2. **Duplicate Detection**\n",
    "   - Pairwise comparison: Compare all entity pairs for duplicates\n",
    "   - Group detection: Find clusters of duplicates using Union-Find algorithm\n",
    "   - Incremental detection: Efficiently detect duplicates between new and existing entities\n",
    "   - Confidence scoring: Multi-factor confidence calculation (similarity + name match + property matches)\n",
    "   - Relationship duplicate detection: Identify duplicate relationships\n",
    "\n",
    "3. **Entity Merging**\n",
    "   - Multiple merge strategies: Keep first, last, most complete, highest confidence, or merge all\n",
    "   - Automatic duplicate detection before merging\n",
    "   - Conflict resolution: Handle property and relationship conflicts\n",
    "   - Provenance preservation: Track which entities were merged\n",
    "   - Merge history: Maintain record of all merge operations\n",
    "   - Quality validation: Validate merged entities for completeness\n",
    "\n",
    "4. **Clustering**\n",
    "   - Graph-based clustering: Union-Find algorithm for connected components\n",
    "   - Hierarchical clustering: Agglomerative clustering for large datasets\n",
    "   - Cluster quality metrics: Cohesion and separation measures\n",
    "   - Incremental updates: Update clusters with new entities\n",
    "\n",
    "5. **Advanced Features**\n",
    "   - Property-specific merge rules: Different strategies for different properties\n",
    "   - Custom conflict resolution: Define custom functions for resolving conflicts\n",
    "   - Method registry: Register and use custom deduplication methods\n",
    "   - Configuration management: Centralized configuration from multiple sources\n",
    "   - Extensibility: Add custom similarity, detection, merge, and clustering methods\n",
    "\n",
    "## Module Architecture\n",
    "\n",
    "### Main Components\n",
    "\n",
    "**Core Classes:**\n",
    "- `DuplicateDetector`: Detects duplicate entities and relationships\n",
    "- `EntityMerger`: Merges duplicate entities with configurable strategies\n",
    "- `SimilarityCalculator`: Calculates multi-factor similarity between entities\n",
    "- `ClusterBuilder`: Builds clusters for efficient batch deduplication\n",
    "- `MergeStrategyManager`: Manages merge strategies and conflict resolution\n",
    "- `MethodRegistry`: Registry for custom deduplication methods\n",
    "- `DeduplicationConfig`: Centralized configuration management\n",
    "\n",
    "**Data Structures:**\n",
    "- `DuplicateCandidate`: Duplicate pair with confidence scores\n",
    "- `DuplicateGroup`: Group of duplicate entities\n",
    "- `MergeOperation`: Merge operation record with metadata\n",
    "- `SimilarityResult`: Similarity calculation result with components\n",
    "- `Cluster`: Entity cluster representation\n",
    "- `ClusterResult`: Cluster building result with quality metrics\n",
    "- `MergeResult`: Merge operation result with conflicts\n",
    "- `MergeStrategy`: Enumeration of merge strategies\n",
    "\n",
    "**Convenience Functions:**\n",
    "- `detect_duplicates()`: Duplicate detection wrapper\n",
    "- `merge_entities()`: Entity merging wrapper\n",
    "- `calculate_similarity()`: Similarity calculation wrapper\n",
    "- `build_clusters()`: Cluster building wrapper\n",
    "- `get_deduplication_method()`: Get method by name\n",
    "- `list_available_methods()`: List all available methods\n",
    "\n",
    "## Algorithms Used\n",
    "\n",
    "**Similarity Calculation:**\n",
    "- Levenshtein Distance: Dynamic programming for edit distance\n",
    "- Jaro Similarity: Character-based similarity with match window\n",
    "- Jaro-Winkler: Jaro with prefix bonus (up to 4 characters)\n",
    "- Cosine Similarity: Vector dot product for embeddings\n",
    "- Jaccard Similarity: Intersection over union for sets\n",
    "- Multi-factor Aggregation: Weighted sum of similarity components\n",
    "\n",
    "**Duplicate Detection:**\n",
    "- Pairwise Comparison: O(n²) all-pairs similarity calculation\n",
    "- Union-Find Algorithm: Disjoint set union for group formation\n",
    "- Confidence Scoring: Multi-factor confidence calculation\n",
    "- Incremental Processing: O(n×m) efficient new vs existing comparison\n",
    "\n",
    "**Clustering:**\n",
    "- Union-Find (DSU): Connected component detection\n",
    "- Hierarchical Clustering: Agglomerative bottom-up clustering\n",
    "- Similarity Graph: Graph construction from similarity scores\n",
    "\n",
    "**Entity Merging:**\n",
    "- Strategy Pattern: Multiple merge strategies\n",
    "- Conflict Resolution: Voting, credibility-weighted, temporal, confidence-based\n",
    "- Property Merging: Rule-based property combination\n",
    "- Provenance Tracking: Metadata preservation during merges\n",
    "\n",
    "## Installation\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "1. [Module Overview](#module-overview)\n",
    "2. [Setup and Sample Data](#setup)\n",
    "3. [Similarity Calculation](#similarity)\n",
    "4. [Duplicate Detection](#detection)\n",
    "5. [Entity Merging](#merging)\n",
    "6. [Clustering](#clustering)\n",
    "7. [Advanced Features](#advanced)\n",
    "8. [Complete Workflow](#workflow)\n",
    "\n",
    "---\n",
    "\n",
    "## Module Overview {#module-overview}\n",
    "\n",
    "The deduplication module provides a complete solution for maintaining clean knowledge graphs by identifying and merging duplicate entities. It supports multiple similarity algorithms, detection methods, merge strategies, and clustering approaches, making it suitable for various use cases from simple exact matching to advanced semantic deduplication.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import all deduplication classes\n",
    "from semantica.deduplication import (\n",
    "    # Main Classes\n",
    "    DuplicateDetector,\n",
    "    EntityMerger,\n",
    "    SimilarityCalculator,\n",
    "    ClusterBuilder,\n",
    "    MergeStrategyManager,\n",
    "    MethodRegistry,\n",
    "    DeduplicationConfig,\n",
    "    # Data Classes\n",
    "    DuplicateCandidate,\n",
    "    DuplicateGroup,\n",
    "    MergeOperation,\n",
    "    SimilarityResult,\n",
    "    Cluster,\n",
    "    ClusterResult,\n",
    "    MergeResult,\n",
    "    MergeStrategy,\n",
    "    # Global Instances\n",
    "    method_registry,\n",
    "    dedup_config,\n",
    ")\n",
    "\n",
    "# Create sample entities with potential duplicates\n",
    "entities = [\n",
    "    {\n",
    "        \"id\": \"e1\",\n",
    "        \"name\": \"Apple Inc.\",\n",
    "        \"type\": \"Company\",\n",
    "        \"founded\": 1976,\n",
    "        \"properties\": {\"industry\": \"Technology\", \"headquarters\": \"Cupertino\"},\n",
    "        \"relationships\": [{\"subject\": \"e1\", \"predicate\": \"founded_by\", \"object\": \"Steve Jobs\"}],\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"e2\",\n",
    "        \"name\": \"Apple Inc\",\n",
    "        \"type\": \"Company\",\n",
    "        \"founded\": 1976,\n",
    "        \"properties\": {\"industry\": \"Tech\", \"headquarters\": \"Cupertino, CA\"},\n",
    "        \"relationships\": [{\"subject\": \"e2\", \"predicate\": \"founded_by\", \"object\": \"Steve Jobs\"}],\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"e3\",\n",
    "        \"name\": \"Microsoft Corporation\",\n",
    "        \"type\": \"Company\",\n",
    "        \"founded\": 1975,\n",
    "        \"properties\": {\"industry\": \"Technology\", \"headquarters\": \"Redmond\"},\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"e4\",\n",
    "        \"name\": \"Microsoft\",\n",
    "        \"type\": \"Company\",\n",
    "        \"founded\": 1975,\n",
    "        \"properties\": {\"industry\": \"Tech\", \"headquarters\": \"Redmond, WA\"},\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"e5\",\n",
    "        \"name\": \"Google LLC\",\n",
    "        \"type\": \"Company\",\n",
    "        \"founded\": 1998,\n",
    "        \"properties\": {\"industry\": \"Technology\"},\n",
    "    },\n",
    "]\n",
    "\n",
    "print(f\"Created {len(entities)} sample entities\")\n",
    "print(\"\\nEntity names:\")\n",
    "for e in entities:\n",
    "    print(f\"  - {e['name']} (ID: {e['id']})\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Similarity Calculation {#similarity}\n",
    "\n",
    "The module provides multiple algorithms for calculating similarity between entities. Similarity is the foundation of duplicate detection.\n",
    "\n",
    "### Available Similarity Methods\n",
    "\n",
    "1. **Exact Matching**: Binary match/no-match for identical strings\n",
    "2. **Levenshtein Distance**: Edit distance between strings (insertions, deletions, substitutions)\n",
    "3. **Jaro-Winkler**: Character-based similarity with prefix bonus for common prefixes\n",
    "4. **Cosine Similarity**: Vector similarity for embeddings using dot product\n",
    "5. **Jaccard Similarity**: Set-based similarity (intersection over union)\n",
    "6. **Property Similarity**: Weighted comparison of property values\n",
    "7. **Relationship Similarity**: Jaccard similarity of relationship sets\n",
    "8. **Multi-factor Similarity**: Weighted aggregation of all components\n",
    "\n",
    "### SimilarityCalculator Class\n",
    "\n",
    "The `SimilarityCalculator` class provides a unified interface for all similarity calculations with configurable weights for different components.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example: Duplicate Detection\n",
    "\n",
    "# Initialize DuplicateDetector\n",
    "detector = DuplicateDetector(\n",
    "    similarity_threshold=0.7,\n",
    "    confidence_threshold=0.6,\n",
    "    use_clustering=True,\n",
    ")\n",
    "\n",
    "# Detect duplicate candidates (pairwise)\n",
    "candidates = detector.detect_duplicates(entities)\n",
    "print(f\"Found {len(candidates)} duplicate candidate(s)\")\n",
    "for candidate in candidates:\n",
    "    print(f\"  {candidate.entity1['name']} <-> {candidate.entity2['name']}\")\n",
    "    print(f\"    Similarity: {candidate.similarity_score:.3f}, Confidence: {candidate.confidence:.3f}\")\n",
    "\n",
    "# Detect duplicate groups\n",
    "duplicate_groups = detector.detect_duplicate_groups(entities)\n",
    "print(f\"\\nFound {len(duplicate_groups)} duplicate group(s)\")\n",
    "for i, group in enumerate(duplicate_groups, 1):\n",
    "    print(f\"  Group {i}: {[e['name'] for e in group.entities]} (confidence: {group.confidence:.3f})\")\n",
    "\n",
    "# Incremental detection\n",
    "existing_entities = entities[:3]\n",
    "new_entities = entities[3:]\n",
    "incremental_candidates = detector.incremental_detect(new_entities, existing_entities, threshold=0.7)\n",
    "print(f\"\\nFound {len(incremental_candidates)} incremental duplicate(s)\")\n",
    "for candidate in incremental_candidates:\n",
    "    print(f\"  {candidate.entity1['name']} duplicates {candidate.entity2['name']} (confidence: {candidate.confidence:.3f})\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Duplicate Detection {#detection}\n",
    "\n",
    "The module detects duplicate entities using similarity metrics and confidence scoring. Detection can be performed pairwise, in groups, or incrementally.\n",
    "\n",
    "### Detection Methods\n",
    "\n",
    "1. **Pairwise Detection**: Compare all entity pairs (O(n²) complexity)\n",
    "2. **Group Detection**: Find clusters of duplicates using Union-Find algorithm\n",
    "3. **Incremental Detection**: Efficiently detect duplicates between new and existing entities (O(n×m))\n",
    "4. **Relationship Detection**: Identify duplicate relationships\n",
    "\n",
    "### Confidence Scoring\n",
    "\n",
    "The module calculates confidence scores using multiple factors:\n",
    "- Similarity score between entities\n",
    "- Name matching (exact or fuzzy)\n",
    "- Property value matches\n",
    "- Entity type matches\n",
    "- Relationship overlap\n",
    "\n",
    "### DuplicateDetector Class\n",
    "\n",
    "The `DuplicateDetector` class provides all duplicate detection capabilities with configurable thresholds and clustering options.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example: Similarity Calculation\n",
    "\n",
    "# Initialize SimilarityCalculator\n",
    "calculator = SimilarityCalculator(\n",
    "    string_weight=0.4,\n",
    "    property_weight=0.3,\n",
    "    relationship_weight=0.2,\n",
    "    embedding_weight=0.1,\n",
    ")\n",
    "\n",
    "# Calculate overall similarity (multi-factor)\n",
    "entity1, entity2 = entities[0], entities[1]\n",
    "result = calculator.calculate_similarity(entity1, entity2)\n",
    "print(f\"Overall Similarity: {result.score:.3f}\")\n",
    "print(f\"Components: {result.components}\")\n",
    "\n",
    "# String similarity methods\n",
    "str1, str2 = \"Apple Inc.\", \"Apple Inc\"\n",
    "for method in [\"levenshtein\", \"jaro_winkler\", \"cosine\"]:\n",
    "    score = calculator.calculate_string_similarity(str1, str2, method=method)\n",
    "    print(f\"{method}: {score:.3f}\")\n",
    "\n",
    "# Property and relationship similarity\n",
    "prop_score = calculator.calculate_property_similarity(entity1, entity2)\n",
    "rel_score = calculator.calculate_relationship_similarity(entity1, entity2)\n",
    "print(f\"\\nProperty Similarity: {prop_score:.3f}\")\n",
    "print(f\"Relationship Similarity: {rel_score:.3f}\")\n",
    "\n",
    "# Batch similarity calculation\n",
    "similarity_pairs = calculator.batch_calculate_similarity(entities, threshold=0.5)\n",
    "print(f\"\\nFound {len(similarity_pairs)} similar pairs (threshold >= 0.5)\")\n",
    "for e1, e2, score in similarity_pairs:\n",
    "    print(f\"  {e1['name']} <-> {e2['name']}: {score:.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Entity Merging {#merging}\n",
    "\n",
    "The module merges duplicate entities into single canonical representations using configurable strategies. Merging preserves provenance, handles conflicts, and maintains merge history.\n",
    "\n",
    "### Merge Strategies\n",
    "\n",
    "1. **KEEP_FIRST**: Preserve the first entity encountered, merge others into it\n",
    "2. **KEEP_LAST**: Preserve the last entity encountered, merge others into it\n",
    "3. **KEEP_MOST_COMPLETE**: Preserve entity with most properties and relationships\n",
    "4. **KEEP_HIGHEST_CONFIDENCE**: Preserve entity with highest confidence score\n",
    "5. **MERGE_ALL**: Create new entity combining all properties and relationships\n",
    "6. **CUSTOM**: User-defined merge logic\n",
    "\n",
    "### Conflict Resolution\n",
    "\n",
    "When merging entities with conflicting property values, the module supports:\n",
    "- Voting: Majority value selection\n",
    "- Credibility-weighted: Weighted by source credibility\n",
    "- Temporal: Most recent value\n",
    "- Confidence-based: Highest confidence value\n",
    "- Custom functions: User-defined resolution logic\n",
    "\n",
    "### EntityMerger Class\n",
    "\n",
    "The `EntityMerger` class provides entity merging with automatic duplicate detection, provenance preservation, and merge history tracking.\n",
    "\n",
    "### MergeStrategyManager Class\n",
    "\n",
    "The `MergeStrategyManager` class provides advanced merge management with property-specific rules and custom conflict resolution functions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example: Entity Merging\n",
    "\n",
    "# Initialize EntityMerger\n",
    "merger = EntityMerger(preserve_provenance=True)\n",
    "\n",
    "# Merge duplicates (automatic detection)\n",
    "merge_operations = merger.merge_duplicates(entities)\n",
    "print(f\"Original entities: {len(entities)}\")\n",
    "print(f\"Merge operations: {len(merge_operations)}\")\n",
    "for i, op in enumerate(merge_operations, 1):\n",
    "    print(f\"  Operation {i}: Merged {len(op.source_entities)} entities → {op.merged_entity.get('name')}\")\n",
    "    if op.merge_result.conflicts:\n",
    "        print(f\"    Conflicts: {len(op.merge_result.conflicts)}\")\n",
    "\n",
    "# Merge with specific strategy\n",
    "operations = merger.merge_duplicates(entities, strategy=MergeStrategy.KEEP_MOST_COMPLETE)\n",
    "print(f\"\\nMerged using KEEP_MOST_COMPLETE: {len(operations)} operations\")\n",
    "\n",
    "# Merge specific group\n",
    "duplicate_entities = [entities[0], entities[1]]\n",
    "operation = merger.merge_entity_group(duplicate_entities, strategy=MergeStrategy.KEEP_FIRST)\n",
    "print(f\"\\nMerged group: {[e['name'] for e in operation.source_entities]} → {operation.merged_entity['name']}\")\n",
    "\n",
    "# Get merge history\n",
    "history = merger.get_merge_history()\n",
    "print(f\"\\nTotal merge operations in history: {len(history)}\")\n",
    "\n",
    "# Validate merge quality\n",
    "if operations:\n",
    "    validation = merger.validate_merge_quality(operations[0])\n",
    "    print(f\"\\nValidation: Valid={validation['valid']}, Quality={validation['quality_score']:.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Clustering {#clustering}\n",
    "\n",
    "The module provides clustering capabilities for efficient batch deduplication of large datasets. Clustering groups similar entities together before deduplication.\n",
    "\n",
    "### Clustering Methods\n",
    "\n",
    "1. **Graph-Based Clustering**: Union-Find algorithm for connected components\n",
    "   - Builds similarity graph from pairwise similarities\n",
    "   - Uses Union-Find (Disjoint Set Union) for efficient component detection\n",
    "   - Suitable for medium-sized datasets\n",
    "\n",
    "2. **Hierarchical Clustering**: Agglomerative bottom-up clustering\n",
    "   - Builds cluster hierarchy by merging similar clusters\n",
    "   - Suitable for large datasets\n",
    "   - Provides cluster quality metrics (cohesion, separation)\n",
    "\n",
    "### Cluster Quality Metrics\n",
    "\n",
    "- **Cohesion**: Average similarity within clusters\n",
    "- **Separation**: Average similarity between clusters\n",
    "- **Cluster Quality Score**: Combined metric for cluster evaluation\n",
    "\n",
    "### ClusterBuilder Class\n",
    "\n",
    "The `ClusterBuilder` class provides clustering with configurable thresholds, quality metrics, and incremental update capabilities.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example: Advanced Merge Strategies\n",
    "\n",
    "# Initialize MergeStrategyManager\n",
    "strategy_manager = MergeStrategyManager(default_strategy=\"keep_most_complete\")\n",
    "\n",
    "# Add property-specific rules\n",
    "strategy_manager.add_property_rule(\"name\", MergeStrategy.KEEP_FIRST, priority=1)\n",
    "strategy_manager.add_property_rule(\"description\", MergeStrategy.MERGE_ALL, priority=1)\n",
    "\n",
    "# Custom conflict resolution\n",
    "def resolve_longest(values):\n",
    "    return max(values, key=len)\n",
    "\n",
    "strategy_manager.add_property_rule(\n",
    "    \"headquarters\", MergeStrategy.CUSTOM, conflict_resolution=resolve_longest, priority=2\n",
    ")\n",
    "\n",
    "print(\"Added merge rules: name=KEEP_FIRST, description=MERGE_ALL, headquarters=CUSTOM\")\n",
    "\n",
    "# Merge entities with property rules\n",
    "duplicate_pair = [entities[0], entities[1]]\n",
    "merge_result = strategy_manager.merge_entities(duplicate_pair)\n",
    "print(f\"\\nMerged Entity: {merge_result.merged_entity.get('name')}\")\n",
    "print(f\"Conflicts: {len(merge_result.conflicts)}\")\n",
    "\n",
    "# Validate merge quality\n",
    "validation = strategy_manager.validate_merge(merge_result)\n",
    "print(f\"Validation: Valid={validation['valid']}, Quality={validation['quality_score']:.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Advanced Features {#advanced}\n",
    "\n",
    "The module provides advanced features for extensibility, configuration, and custom method registration.\n",
    "\n",
    "### Configuration Management\n",
    "\n",
    "The `DeduplicationConfig` class provides centralized configuration from multiple sources:\n",
    "- Programmatic configuration (via `set()` method)\n",
    "- Environment variables\n",
    "- Configuration files (YAML, JSON, TOML)\n",
    "- Default values\n",
    "\n",
    "Configuration can be set globally or per-method for fine-grained control.\n",
    "\n",
    "### Method Registry\n",
    "\n",
    "The `MethodRegistry` class allows registration of custom deduplication methods:\n",
    "- Custom similarity calculation methods\n",
    "- Custom duplicate detection methods\n",
    "- Custom entity merging methods\n",
    "- Custom clustering methods\n",
    "\n",
    "Registered methods can be used throughout the module via the registry.\n",
    "\n",
    "### Property-Specific Merge Rules\n",
    "\n",
    "The `MergeStrategyManager` supports property-specific merge rules:\n",
    "- Different merge strategies for different properties\n",
    "- Custom conflict resolution functions per property\n",
    "- Priority-based rule application\n",
    "- Flexible rule composition\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example: Clustering\n",
    "\n",
    "# Initialize ClusterBuilder\n",
    "cluster_builder = ClusterBuilder(\n",
    "    similarity_threshold=0.7,\n",
    "    min_cluster_size=2,\n",
    "    max_cluster_size=100,\n",
    "    use_hierarchical=False,\n",
    ")\n",
    "\n",
    "# Graph-based clustering\n",
    "cluster_result = cluster_builder.build_clusters(entities)\n",
    "print(f\"Clusters: {len(cluster_result.clusters)}, Unclustered: {len(cluster_result.unclustered)}\")\n",
    "print(f\"Quality Metrics: {cluster_result.quality_metrics}\")\n",
    "for cluster in cluster_result.clusters:\n",
    "    print(f\"  {cluster.cluster_id}: {len(cluster.entities)} entities (quality: {cluster.quality_score:.3f})\")\n",
    "\n",
    "# Hierarchical clustering\n",
    "hierarchical_builder = ClusterBuilder(similarity_threshold=0.7, use_hierarchical=True)\n",
    "hierarchical_result = hierarchical_builder.build_clusters(entities)\n",
    "print(f\"\\nHierarchical Clustering: {len(hierarchical_result.clusters)} clusters\")\n",
    "print(f\"Quality Metrics: {hierarchical_result.quality_metrics}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Complete Workflow {#workflow}\n",
    "\n",
    "A typical deduplication workflow involves:\n",
    "1. Configuration: Set similarity thresholds and method preferences\n",
    "2. Clustering: Build clusters of similar entities (optional, for large datasets)\n",
    "3. Detection: Identify duplicate entities within clusters or entire dataset\n",
    "4. Merging: Merge duplicate entities using appropriate strategies\n",
    "5. Validation: Validate merge quality and completeness\n",
    "6. History: Track merge operations for audit and rollback\n",
    "\n",
    "The module provides both class-based and function-based interfaces for flexibility.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example: Configuration and Method Registry\n",
    "\n",
    "# Configuration management\n",
    "threshold = dedup_config.get(\"similarity_threshold\", default=0.7)\n",
    "confidence = dedup_config.get(\"confidence_threshold\", default=0.6)\n",
    "print(f\"Current: similarity_threshold={threshold}, confidence_threshold={confidence}\")\n",
    "\n",
    "# Set configuration programmatically\n",
    "dedup_config.set(\"similarity_threshold\", 0.8)\n",
    "dedup_config.set(\"confidence_threshold\", 0.7)\n",
    "print(f\"Updated: similarity_threshold={dedup_config.get('similarity_threshold')}\")\n",
    "\n",
    "# Method-specific configuration\n",
    "dedup_config.set_method_config(\"levenshtein\", case_sensitive=False)\n",
    "levenshtein_config = dedup_config.get_method_config(\"levenshtein\")\n",
    "print(f\"Method config (levenshtein): {levenshtein_config}\")\n",
    "\n",
    "# Custom method registration\n",
    "def word_overlap_similarity(entity1, entity2, **kwargs):\n",
    "    \"\"\"Custom similarity based on word overlap.\"\"\"\n",
    "    name1 = entity1.get(\"name\", \"\").lower().split()\n",
    "    name2 = entity2.get(\"name\", \"\").lower().split()\n",
    "    \n",
    "    if not name1 or not name2:\n",
    "        return SimilarityResult(score=0.0, method=\"word_overlap\")\n",
    "    \n",
    "    set1, set2 = set(name1), set(name2)\n",
    "    intersection = len(set1 & set2)\n",
    "    union = len(set1 | set2)\n",
    "    score = intersection / union if union > 0 else 0.0\n",
    "    return SimilarityResult(score=score, method=\"word_overlap\")\n",
    "\n",
    "# Register custom method\n",
    "method_registry.register(\"similarity\", \"word_overlap\", word_overlap_similarity)\n",
    "print(\"\\nRegistered 'word_overlap' similarity method\")\n",
    "\n",
    "# Use custom method\n",
    "custom_method = method_registry.get(\"similarity\", \"word_overlap\")\n",
    "if custom_method:\n",
    "    result = custom_method(entities[0], entities[1])\n",
    "    print(f\"Word Overlap Similarity: {result.score:.3f}\")\n",
    "\n",
    "# List all registered methods\n",
    "all_registered = method_registry.list_all()\n",
    "print(f\"\\nRegistered Methods: {all_registered}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Practical Examples\n",
    "\n",
    "The following sections demonstrate practical usage of the module components.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example: Complete Workflow\n",
    "\n",
    "# Complete deduplication workflow\n",
    "\n",
    "# Step 1: Configure\n",
    "dedup_config.set(\"similarity_threshold\", 0.75)\n",
    "dedup_config.set(\"confidence_threshold\", 0.65)\n",
    "\n",
    "# Step 2: Build clusters\n",
    "cluster_builder = ClusterBuilder(similarity_threshold=0.75, min_cluster_size=2, max_cluster_size=50)\n",
    "cluster_result = cluster_builder.build_clusters(entities)\n",
    "print(f\"Step 1: Created {len(cluster_result.clusters)} clusters\")\n",
    "\n",
    "# Step 3: Detect duplicates\n",
    "detector = DuplicateDetector(similarity_threshold=0.75, confidence_threshold=0.65)\n",
    "all_duplicate_groups = []\n",
    "for cluster in cluster_result.clusters:\n",
    "    groups = detector.detect_duplicate_groups(cluster.entities)\n",
    "    all_duplicate_groups.extend(groups)\n",
    "print(f\"Step 2: Found {len(all_duplicate_groups)} duplicate groups\")\n",
    "\n",
    "# Step 4: Merge duplicates\n",
    "merger = EntityMerger(preserve_provenance=True)\n",
    "merge_operations = merger.merge_duplicates(entities, strategy=MergeStrategy.KEEP_MOST_COMPLETE)\n",
    "print(f\"Step 3: Performed {len(merge_operations)} merge operations\")\n",
    "\n",
    "# Step 5: Extract results\n",
    "merged_entities = [op.merged_entity for op in merge_operations]\n",
    "print(f\"\\nResults:\")\n",
    "print(f\"  Original: {len(entities)} entities\")\n",
    "print(f\"  Merged: {len(merged_entities)} entities\")\n",
    "print(f\"  Reduction: {len(entities) - len(merged_entities)} entities\")\n",
    "\n",
    "# Step 6: Validate merge quality\n",
    "for i, op in enumerate(merge_operations, 1):\n",
    "    validation = merger.validate_merge_quality(op)\n",
    "    print(f\"  Merge {i}: Valid={validation['valid']}, Quality={validation['quality_score']:.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Similarity Calculation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Complete deduplication workflow\n",
    "\n",
    "# Step 1: Configure\n",
    "dedup_config.set(\"similarity_threshold\", 0.75)\n",
    "dedup_config.set(\"confidence_threshold\", 0.65)\n",
    "\n",
    "# Step 2: Build clusters\n",
    "cluster_builder = ClusterBuilder(similarity_threshold=0.75, min_cluster_size=2, max_cluster_size=50)\n",
    "cluster_result = cluster_builder.build_clusters(entities)\n",
    "print(f\"Step 1: Created {len(cluster_result.clusters)} clusters\")\n",
    "\n",
    "# Step 3: Detect duplicates\n",
    "detector = DuplicateDetector(similarity_threshold=0.75, confidence_threshold=0.65)\n",
    "all_duplicate_groups = []\n",
    "for cluster in cluster_result.clusters:\n",
    "    groups = detector.detect_duplicate_groups(cluster.entities)\n",
    "    all_duplicate_groups.extend(groups)\n",
    "print(f\"Step 2: Found {len(all_duplicate_groups)} duplicate groups\")\n",
    "\n",
    "# Step 4: Merge duplicates\n",
    "merger = EntityMerger(preserve_provenance=True)\n",
    "merge_operations = merger.merge_duplicates(entities, strategy=MergeStrategy.KEEP_MOST_COMPLETE)\n",
    "print(f\"Step 3: Performed {len(merge_operations)} merge operations\")\n",
    "\n",
    "# Step 5: Extract results\n",
    "merged_entities = [op.merged_entity for op in merge_operations]\n",
    "print(f\"\\nResults:\")\n",
    "print(f\"  Original: {len(entities)} entities\")\n",
    "print(f\"  Merged: {len(merged_entities)} entities\")\n",
    "print(f\"  Reduction: {len(entities) - len(merged_entities)} entities\")\n",
    "\n",
    "# Step 6: Validate merge quality\n",
    "for i, op in enumerate(merge_operations, 1):\n",
    "    validation = merger.validate_merge_quality(op)\n",
    "    print(f\"  Merge {i}: Valid={validation['valid']}, Quality={validation['quality_score']:.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Duplicate Detection\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
