---
title: "Deduplication Module"
description: "Entity deduplication v1/v2 — similarity scoring, blocking, merging, and cluster-based batch processing."
icon: "copy"
---
`semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control.
## Exported Classes
```python
from semantica.deduplication import (
DuplicateDetector, # pairwise + batch duplicate detection
EntityMerger, # merge duplicate groups with per-property policies
SimilarityCalculator, # Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding
ClusterBuilder, # Union-Find + hierarchical clustering
PropertyMergeRule, # enum: KEEP_FIRST, KEEP_LONGEST, UNION, VOTING, ...
MergeStrategyManager, # manage and apply merge strategies
# Convenience functions
detect_duplicates, # quick: detect_duplicates(entities, method="semantic_v2")
merge_entities, # quick: merge_entities(entities, duplicates, method="union")
calculate_similarity, # quick: calculate_similarity(a, b, method="hybrid_v2")
# Registry
method_registry, # register custom similarity functions
)
```
## What You Get
- **`DuplicateDetector`** — pairwise and batch duplicate detection with configurable strategies
- **`EntityMerger`** — merge duplicate groups with configurable property-level merge policies
- **`SimilarityCalculator`** — multi-factor similarity: Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding
- **`ClusterBuilder`** — Union-Find and hierarchical clustering for large-scale batch deduplication
- **Convenience functions** — `detect_duplicates`, `merge_entities`, `calculate_similarity`
## DuplicateDetector
Find duplicate entity pairs with configurable strategies and result filtering:
```python
from semantica.deduplication import DuplicateDetector
detector = DuplicateDetector(similarity_threshold=0.85)
duplicates = detector.detect_duplicates(entities)
for dup in duplicates:
print(f"{dup.entity_a} ≈ {dup.entity_b} ({dup.similarity:.2f})")
```
Fine-grained control over strategy, thresholds, and result size:
```python
duplicates = detector.detect_duplicates(
entities,
strategy="semantic_v2", # see strategies table below
min_similarity=0.85, # minimum score to consider a match
top_k_per_entity=3, # max candidates per entity
max_results=100, # total result cap
sort_by="similarity", # "similarity" | "entity_id" | "cluster_size"
)
```
### Detection Strategies
| Strategy | Algorithm | Speed | Accuracy |
| -------- | --------- | ----- | -------- |
| `jaro_winkler` | String similarity (v1) | Fast | Medium |
| `blocking_v2` | Blocking + Jaro-Winkler (v2) | Very fast | Medium |
| `hybrid_v2` | Blocking + semantic + string (v2) | Fast | High |
| `semantic_v2` | Embedding similarity (v2) | Medium | Highest |
**v0.5.0 fix:** `DuplicateDetector` no longer produces duplicate definition errors when the same entity appears in multiple sources with identical definitions.
## EntityMerger
Merges detected duplicate groups into canonical entities, preserving provenance:
```python
from semantica.deduplication import EntityMerger
merger = EntityMerger()
merged_entities = merger.merge_duplicates(
entities,
strategy="keep_most_complete", # see strategies table below
preserve_provenance=True, # keep source references after merge
)
```
### Merge Strategies
| Strategy | Behavior |
| -------- | -------- |
| `keep_first` | Keep the first entity in each duplicate group |
| `keep_last` | Keep the most recently seen entity |
| `keep_most_complete` | Keep the entity with the most non-null properties |
| `union` | Merge all properties — non-conflicting fields combined |
| `voting` | Most common property value wins |
Fine-grained per-property merge rules:
```python
from semantica.deduplication import EntityMerger, PropertyMergeRule
merger = EntityMerger(
property_rules={
"name": PropertyMergeRule.KEEP_FIRST,
"aliases": PropertyMergeRule.UNION,
"description": PropertyMergeRule.KEEP_LONGEST,
}
)
```
## SimilarityCalculator
Compute multi-factor similarity scores between entity pairs:
```python
from semantica.deduplication import SimilarityCalculator
calc = SimilarityCalculator()
score = calc.calculate_similarity(entity_a, entity_b)
# → SimilarityResult(score=0.91, components={...})
print(score.score) # overall score 0.0–1.0
print(score.components["label"]) # label similarity
print(score.components["embedding"]) # semantic similarity
print(score.components["property"]) # property overlap
```
Individual string and vector metrics:
```python
lev = calc.levenshtein("Apple Inc.", "Apple Inc")
jaro = calc.jaro_winkler("Steve Jobs", "Steven Jobs")
cos = calc.cosine_similarity(embedding_a, embedding_b)
jacc = calc.jaccard({"founded", "tech"}, {"founded", "technology"})
```
## ClusterBuilder
Build entity clusters for large-scale batch deduplication:
```python
from semantica.deduplication import ClusterBuilder
builder = ClusterBuilder(algorithm="union_find") # or "hierarchical"
result = builder.build_clusters(entities, similarity_threshold=0.85)
print(f"Clusters: {len(result.clusters)}")
for cluster in result.clusters:
print(f" [{cluster.id}] {cluster.members} — cohesion: {cluster.cohesion:.2f}")
```
## Blocking Strategies
Blocking reduces the O(n²) pairwise comparison problem to a manageable candidate set:
```python
detector = DuplicateDetector(
blocking_strategy="token", # "token" | "phonetic" | "ngram"
blocking_threshold=0.6,
similarity_threshold=0.85
)
```
## Custom Similarity Functions
Register domain-specific similarity logic:
```python
from semantica.deduplication import method_registry
def drug_name_similarity(entity_a, entity_b):
# Match drug names by active compound
return score # 0.0 to 1.0
method_registry.register("similarity", "drug_name", drug_name_similarity)
detector = DuplicateDetector(similarity_method="drug_name")
```
## Convenience Functions
```python
from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity
# Quick detection
duplicates = detect_duplicates(entities, method="semantic_v2", similarity_threshold=0.85)
# Quick merge
merged = merge_entities(entities, duplicates, method="keep_most_complete")
# Quick similarity
score = calculate_similarity(entity_a, entity_b, method="hybrid_v2")
```
Detect value conflicts between non-duplicate entities.
GraphBuilder uses deduplication during construction.
Normalize entity names before deduplication.
Track merged entity lineage.