--- title: "Deduplication Module" description: "Entity deduplication v1/v2 — similarity scoring, blocking, merging, and cluster-based batch processing." icon: "copy" --- `semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control. ## Why Deduplicate? Real-world data sources disagree on names. "Apple Inc.", "Apple Computer", and "Apple International Ltd" can all refer to the same company — but if they land in your knowledge graph as separate nodes, every query that should return one entity returns three. Relationships, analytics, and retrieval all degrade. Deduplication solves this before it reaches the graph: - **Cross-source ingestion** — Wikipedia calls it "OpenAI", SEC filings call it "OpenAI, Inc.", your CRM calls it "OpenAI LLC" - **Data entry variation** — "Steve Jobs", "Steven P. Jobs", "S. Jobs" are the same person - **Transliteration differences** — Cyrillic, Chinese, or Arabic names romanized inconsistently across sources - **Abbreviation drift** — "US", "U.S.", "United States", "USA" all mean the same thing ## What You Get Pairwise and batch duplicate detection with configurable strategies and result filtering. Merge duplicate groups with configurable property-level merge policies. Multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, and embedding. Union-Find and hierarchical clustering for large-scale batch deduplication. Reusable per-property merge rule configurations — define once, apply across operations. `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster than v1 equivalents. ## Quick Start ```python from semantica.deduplication import detect_duplicates, merge_entities # Detect duplicates = detect_duplicates(entities, method="hybrid_v2", similarity_threshold=0.85) # Merge merged = merge_entities(entities, duplicates, method="keep_most_complete") print(f"Reduced {len(entities)} → {len(merged)} entities") ``` ## Choosing a Strategy Combines blocking + string similarity + semantic embedding — best default for production: ```python from semantica.deduplication import DuplicateDetector detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") ``` Best for: general production use — handles 95% of real-world cases without GPU. Catches string variants ("Apple Inc." / "Apple Inc") and semantic aliases ("Machine Learning" / "ML"). Pure embedding-based similarity — highest accuracy for cross-language and abbreviation matching: ```python detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities, strategy="semantic_v2") ``` Best for: entities that use different words for the same concept ("ML" vs "Machine Learning"), cross-language entity matching, and abbreviation expansion. Requires embedding model. Blocking + Jaro-Winkler string similarity only — fastest option for CPU-only environments: ```python detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities, strategy="blocking_v2") ``` Best for: large datasets (500K+ entities) where speed is critical and entities are in the same language. No GPU required. | Strategy | Algorithm | Speed | Accuracy | Best For | | -------- | --------- | ----- | -------- | -------- | | `jaro_winkler` | String similarity only (v1) | Fast | Medium | Small datasets, names, single-source | | `blocking_v2` | Blocking + Jaro-Winkler | Very fast | Medium | Large datasets, speed-critical, CPU-only | | `hybrid_v2` | Blocking + string + semantic | Fast | High | General production use — best default | | `semantic_v2` | Embedding similarity | Medium | Highest | Semantic aliases, cross-language, abbreviations | **Rules of thumb:** - Start with `hybrid_v2` — it handles 95% of real-world cases without GPU - Use `semantic_v2` when entities use different words for the same concept - Use `blocking_v2` when processing >500k entities and speed matters most - Never use v1 strategies on new projects — they exist for backwards compatibility only ### Threshold Tuning | Domain | Recommended Threshold | Notes | | ------ | --------------------- | ----- | | Person names | 0.85–0.90 | Names vary a lot; too high misses "Steve" / "Steven" | | Organization names | 0.80–0.88 | Corporate suffixes create variation; lower threshold helps | | Product names | 0.88–0.95 | Product names are more stable | | Medical terms | 0.90–0.95 | High precision required; false merges are dangerous | | General entities | 0.85 | Safe default starting point | Start at 0.85, inspect false positives and false negatives, then adjust ±0.05. ## DuplicateDetector **v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions. ```python from semantica.deduplication import DuplicateDetector detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities) for dup in duplicates: print(f"{dup.entity_a} ≈ {dup.entity_b} ({dup.similarity:.2f})") ``` Fine-grained control over strategy, thresholds, and result size: ```python duplicates = detector.detect_duplicates( entities, strategy="hybrid_v2", min_similarity=0.85, top_k_per_entity=3, # max candidates per entity — avoids false-positive floods max_results=100, sort_by="similarity", # "similarity" | "entity_id" | "cluster_size" ) ``` **Key behaviours:** - Defaults to `hybrid_v2` when no strategy is specified - `top_k_per_entity=3` prevents one entity from flooding results by being a near-match to everything - Pairs are returned once — never both `(A, B)` and `(B, A)` - `sort_by="similarity"` puts highest-confidence duplicates first for faster manual review ## EntityMerger Merge detected duplicate groups into canonical entities, preserving provenance: ```python from semantica.deduplication import EntityMerger merger = EntityMerger() merged_entities = merger.merge_duplicates( entities, strategy="keep_most_complete", preserve_provenance=True, ) print(f"Merged to: {len(merged_entities)} canonical entities") ``` ### Merge Strategies | Strategy | Behavior | When to Use | | -------- | -------- | ----------- | | `keep_first` | Keep the first entity in each group | Source order is meaningful (most authoritative first) | | `keep_last` | Keep the most recently seen entity | Most recent source is most accurate | | `keep_most_complete` | Keep the entity with the most non-null properties | Default — maximizes data richness | | `union` | Merge all properties; combine non-conflicting fields | You want every known alias, tag, and label | | `voting` | Most common property value wins | Multiple semi-reliable sources | ### Per-Property Merge Rules ```python from semantica.deduplication import EntityMerger, PropertyMergeRule merger = EntityMerger( property_rules={ "name": PropertyMergeRule.KEEP_FIRST, "aliases": PropertyMergeRule.UNION, "description": PropertyMergeRule.KEEP_LONGEST, "confidence": PropertyMergeRule.MAX, "created_at": PropertyMergeRule.KEEP_FIRST, "updated_at": PropertyMergeRule.KEEP_LAST, } ) merged_entities = merger.merge_duplicates(entities, preserve_provenance=True) ``` | Rule | Behaviour | | ---- | --------- | | `KEEP_FIRST` | Value from the first entity in the group | | `KEEP_LAST` | Value from the last entity | | `KEEP_LONGEST` | Longest non-null string value | | `KEEP_MOST_COMPLETE` | Entity with the most non-null fields (default fallback) | | `UNION` | Combine all unique values into a list | | `MAX` | Numerically largest value | | `MIN` | Numerically smallest value | | `VOTING` | Most frequently occurring value | ## SimilarityCalculator Compute multi-factor similarity scores — useful for debugging why two entities were (or were not) detected as duplicates: ```python from semantica.deduplication import SimilarityCalculator calc = SimilarityCalculator() score = calc.calculate_similarity(entity_a, entity_b) print(score.score) # overall score 0.0–1.0 print(score.components["label"]) # label similarity contribution print(score.components["embedding"]) # semantic similarity contribution print(score.components["property"]) # property overlap contribution lev = calc.levenshtein("Apple Inc.", "Apple Inc") jaro = calc.jaro_winkler("Steve Jobs", "Steven Jobs") cos = calc.cosine_similarity(embedding_a, embedding_b) jacc = calc.jaccard({"founded", "tech"}, {"founded", "technology"}) ``` ## ClusterBuilder Build entity clusters for large-scale batch deduplication: ```python from semantica.deduplication import ClusterBuilder builder = ClusterBuilder(algorithm="union_find") # or "hierarchical" result = builder.build_clusters(entities, similarity_threshold=0.85) print(f"Original: {len(entities)} entities") print(f"Clusters: {len(result.clusters)} groups") print(f"Singletons: {result.singleton_count}") for cluster in result.clusters: print(f" [{cluster.id}] members={cluster.members} cohesion={cluster.cohesion:.2f}") ``` ### Union-Find vs Hierarchical ```python # Union-Find — O(n·α(n)), scales to millions; use for production builder = ClusterBuilder(algorithm="union_find") # Hierarchical — tighter clusters; use when quality matters more than speed builder = ClusterBuilder(algorithm="hierarchical", linkage="average") result = builder.build_clusters(entities, similarity_threshold=0.85) ``` **Key behaviours:** - Union-Find groups entities transitively: if A≈B and B≈C, all three land in one cluster even if A and C are only 0.70 similar - Hierarchical with `linkage="average"` avoids chaining by requiring average similarity across all pairs to meet the threshold ### Cluster Quality Metrics ```python print(f"Silhouette score: {result.quality.silhouette_score:.3f}") # → -1.0 to 1.0; above 0.5 is good, above 0.7 is excellent print(f"Avg cohesion: {result.quality.avg_cohesion:.3f}") print(f"Avg separation: {result.quality.avg_separation:.3f}") ``` ## MergeStrategyManager Define complex, reusable merge configurations once and apply them across multiple operations: ```python from semantica.deduplication import MergeStrategyManager, PropertyMergeRule manager = MergeStrategyManager() manager.add_rule("name", PropertyMergeRule.KEEP_FIRST) manager.add_rule("aliases", PropertyMergeRule.UNION) manager.add_rule("description", PropertyMergeRule.KEEP_LONGEST) manager.add_rule("confidence", PropertyMergeRule.MAX) manager.add_rule("sources", PropertyMergeRule.UNION) manager.add_rule("created_at", PropertyMergeRule.KEEP_FIRST) manager.add_rule("updated_at", PropertyMergeRule.KEEP_LAST) manager.set_default_rule(PropertyMergeRule.KEEP_MOST_COMPLETE) merged_entity = manager.merge(duplicate_group) ``` ## Blocking Strategies ```python detector = DuplicateDetector( blocking_strategy="token", # "token" | "phonetic" | "ngram" blocking_threshold=0.6, similarity_threshold=0.85 ) ``` | Blocking Strategy | How It Works | Best For | | ----------------- | ------------ | -------- | | `token` | Shared token overlap (default) | General entity names | | `phonetic` | Soundex/Metaphone phonetic codes | Names with spelling variations | | `ngram` | Character n-gram overlap | Short strings, typos | ## Custom Similarity Functions ```python from semantica.deduplication import method_registry def drug_name_similarity(entity_a, entity_b) -> float: compound_a = entity_a.properties.get("active_compound", "") compound_b = entity_b.properties.get("active_compound", "") return 1.0 if compound_a == compound_b else 0.0 method_registry.register("similarity", "drug_name", drug_name_similarity) detector = DuplicateDetector(similarity_method="drug_name", similarity_threshold=0.90) ``` ## Schemas ```python @dataclass class Cluster: id: str members: List[str] # entity IDs in this duplicate group cohesion: float # mean pairwise similarity within the cluster (0–1) centroid: str # member ID closest to the cluster centroid @dataclass class ClusterResult: clusters: List[Cluster] singleton_count: int # entities with no duplicates found merge_candidates: int # clusters with > 1 member quality: ClusterQuality ``` ```python @dataclass class ClusterQuality: silhouette_score: float # -1 to 1; above 0.5 is good avg_cohesion: float # mean within-cluster similarity avg_separation: float # mean between-cluster distance ``` ## End-to-End Pipeline ```python entities = load_entities_from_sources(["crunchbase", "wikipedia", "internal_db"]) print(f"Loaded: {len(entities)} raw entities") ``` ```python from semantica.deduplication import DuplicateDetector detector = DuplicateDetector(similarity_threshold=0.85) duplicates = detector.detect_duplicates(entities, strategy="hybrid_v2") print(f"Found: {len(duplicates)} duplicate pairs") ``` ```python from semantica.deduplication import MergeStrategyManager, PropertyMergeRule manager = MergeStrategyManager() manager.add_rule("name", PropertyMergeRule.KEEP_FIRST) manager.add_rule("aliases", PropertyMergeRule.UNION) manager.add_rule("description", PropertyMergeRule.KEEP_LONGEST) manager.set_default_rule(PropertyMergeRule.KEEP_MOST_COMPLETE) ``` ```python from semantica.deduplication import EntityMerger merger = EntityMerger() merged = merger.merge_duplicates(entities, preserve_provenance=True) print(f"Result: {len(merged)} canonical entities") for entity in merged: if hasattr(entity, "source_entities"): print(f"{entity.label} merged from: {entity.source_entities}") ``` ## Tips and Common Pitfalls **Normalize before deduplicating.** Run `TextNormalizer` and `EntityNormalizer` first. "APPLE INC." and "apple inc." will score 0.50 on string similarity but 1.0 after case normalization. See the [Normalize](normalize) module. **Too many false positives?** Raise `similarity_threshold` by 0.05 or switch from `blocking_v2` to `hybrid_v2` to add semantic precision on top of string matching. **Too many missed duplicates?** Lower `similarity_threshold` by 0.05, or switch to `semantic_v2` to catch entities that use different words ("ML" vs "Machine Learning"). **Union-Find over-merges?** The transitive grouping means weak chains can connect unrelated entities. Switch to `hierarchical` clustering with `linkage="average"` to require that every pair within a cluster meets the threshold — not just a chain of nearby pairs. **Preserve provenance on merge.** Set `preserve_provenance=True` so you can always trace which source contributed each property to the merged entity. Critical for audit and debugging. **Inspect similarity components.** When a pair is flagged and you're not sure why, use `SimilarityCalculator.calculate_similarity()` to see the per-component breakdown (`label`, `embedding`, `property`) and identify which factor is driving the match. **Don't deduplicate after building the graph.** Deduplicate entities *before* `GraphBuilder` ingests them. Merging inside a live graph is possible but requires tracking and rewriting all relationship endpoints. Detect value conflicts between non-duplicate entities. GraphBuilder uses deduplication during construction. Normalize entity names before deduplication for better accuracy. Track merged entity lineage and source attribution.