semantica/docs/reference/deduplication.md

Deduplication

Advanced entity deduplication and resolution system for maintaining a clean, single-source-of-truth Knowledge Graph.

---

🎯 Overview

• :material-content-duplicate:{ .lg .middle } Duplicate Detection

  ---

  Identify duplicates using multi-factor similarity metrics

• :material-set-merge:{ .lg .middle } Entity Merging

  ---

  Merge entities with configurable strategies (Keep First, Most Complete, etc.)

• :material-group:{ .lg .middle } Clustering

  ---

  Cluster similar entities for efficient batch processing

• :material-calculator:{ .lg .middle } Similarity Metrics

  ---

  Levenshtein, Jaro-Winkler, Cosine, and Jaccard similarity support

• :material-history:{ .lg .middle } Provenance

  ---

  Preserve data lineage and history during merges

• :material-scale:{ .lg .middle } Scalable

  ---

  Batch processing and blocking for large datasets

!!! tip "When to Use" - Data Ingestion: Clean incoming data before adding to the graph - Graph Maintenance: Periodically clean up existing knowledge graphs - Entity Resolution: Resolve entities from different sources (e.g., "Apple" vs "Apple Inc.")

---

⚙️ Algorithms Used

Similarity Calculation

• Levenshtein Distance: Edit distance for string difference
• Jaro-Winkler: String similarity with prefix weighting (good for names)
• Cosine Similarity: Vector similarity for embeddings
• Jaccard Similarity: Set overlap for properties/relationships
• Multi-factor Aggregation: Weighted sum of multiple metrics

Duplicate Detection

• Pairwise Comparison: O(n²) comparison (for small sets)
• Blocking/Indexing: Reduce search space for large sets
• Union-Find: Disjoint set data structure for grouping duplicates
• Confidence Scoring: 0.0 - 1.0 probability score for duplicates

Clustering

• Hierarchical Clustering: Agglomerative bottom-up clustering
• Connected Components: Graph-based cluster detection
• Cluster Quality: Cohesion and separation metrics

Entity Merging

• Strategy Pattern: Pluggable merge logic
• Property Union: Combining unique properties
• Relationship Merging: Re-linking relationships to the merged entity

---

Main Classes

DuplicateDetector

Identifies potential duplicates in a dataset.

Methods:

Method	Description	Algorithm
detect_duplicates(entities)	Find duplicate pairs	Pairwise/Blocking
detect_duplicate_groups(entities)	Find clusters of duplicates	Union-Find

Example:

from semantica.deduplication import DuplicateDetector

detector = DuplicateDetector(similarity_threshold=0.85)
duplicates = detector.detect_duplicates(entities)

for group in duplicates:
    print(f"Found group of {len(group)} duplicates")

EntityMerger

Merges duplicate entities into a single canonical entity.

Methods:

Method	Description	Strategy
merge_duplicates(entities)	Execute merge	Configured Strategy
merge_group(group)	Merge specific group	Configured Strategy

Strategies:

• KEEP_FIRST: Keep the first entity encountered
• KEEP_MOST_COMPLETE: Keep entity with most properties
• KEEP_HIGHEST_CONFIDENCE: Keep entity with highest confidence score
• MERGE_ALL: Create new entity combining all info

Example:

from semantica.deduplication import EntityMerger

merger = EntityMerger(strategy="keep_most_complete")
result = merger.merge_duplicates(entities)

SimilarityCalculator

Calculates similarity between entities.

Methods:

Method	Description
calculate(e1, e2)	Get aggregate score
string_similarity(s1, s2)	Text comparison

ClusterBuilder

Builds clusters for batch processing.

Methods:

Method	Description	Algorithm
build_clusters(entities)	Create clusters	Hierarchical/Graph

---

Convenience Functions

from semantica.deduplication import deduplicate

# Detect and merge in one step
result = deduplicate(
    entities,
    similarity_threshold=0.8,
    merge_strategy="keep_most_complete"
)

print(f"Reduced {result['statistics']['reduction']} entities")

---

Configuration

Environment Variables

export DEDUP_SIMILARITY_THRESHOLD=0.8
export DEDUP_MERGE_STRATEGY=keep_most_complete
export DEDUP_BLOCKING_ENABLED=true

YAML Configuration

deduplication:
  thresholds:
    similarity: 0.8
    confidence: 0.7
    
  weights:
    name: 0.6
    type: 0.2
    attributes: 0.2
    
  blocking:
    enabled: true
    method: "token_blocking"

---

Integration Examples

Ingestion Pipeline

from semantica.ingest import Ingestor
from semantica.deduplication import deduplicate
from semantica.kg import KnowledgeGraph

# 1. Ingest
ingestor = Ingestor()
raw_entities = ingestor.ingest_batch(files)

# 2. Deduplicate
dedup_result = deduplicate(
    raw_entities,
    similarity_threshold=0.85,
    merge_strategy="merge_all"
)

# 3. Load to KG
kg = KnowledgeGraph()
kg.add_entities(dedup_result['merged_entities'])

---

Best Practices

1. Block First: For >1000 entities, enable blocking to avoid O(n²) performance.
2. Tune Thresholds: Start with 0.85 and adjust based on false positive/negative rates.
3. Preserve Provenance: Keep preserve_provenance=True to track where merged data came from.
4. Normalize: Run normalize module before deduplication for best results.

---

Troubleshooting

Issue: Merging "Apple" and "Apple Pie" (False Positive). Solution: Increase threshold or use Jaro-Winkler which penalizes prefix mismatches.

detector = DuplicateDetector(
    similarity_method="jaro_winkler",
    similarity_threshold=0.9
)

Issue: Slow performance on large datasets. Solution: Use ClusterBuilder with blocking.

---

See Also

• Conflicts Module - Handling conflicting values during merge
• Normalize Module - Pre-processing for better matching
• Knowledge Graph Module - Target for deduplicated data