mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
KaifAhmad1
7a6f1d0417
Add a Cite Us section to the README with BibTeX citation info, and align it with docs/citation.md (author/organization: Semantica, 2026). Update LICENSE and docs/project-license.md copyright holder to Semantica, and replace the stale Hawksight-AI GitHub org slug with semantica-agi across READMEs, plugin manifests, cookbook notebooks, and GitHub templates.
378 lines
13 KiB
Plaintext
378 lines
13 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/18_Deduplication.ipynb)\n",
|
|
"\n",
|
|
"# Deduplication in Semantica\n",
|
|
"\n",
|
|
"Welcome to the **Deduplication** walkthrough! In any Knowledge Graph, data often comes from multiple sources, leading to duplicate entities (e.g., \"Apple Inc.\" vs. \"Apple Inc\"). \n",
|
|
"\n",
|
|
"Semantica provides a robust **Deduplication Module** to help you:\n",
|
|
"1. **Calculate Similarity**: Compare entities using strings, properties, and embeddings.\n",
|
|
"2. **Detect Duplicates**: Find pairs or groups of entities that represent the same real-world object.\n",
|
|
"3. **Cluster Entities**: Group similar entities together.\n",
|
|
"4. **Merge Entities**: Combine duplicates into a single, canonical entity while resolving conflicts.\n",
|
|
"\n",
|
|
"This notebook will guide you through each step with clear examples."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Install Semantica\n",
|
|
"!pip install -q semantica"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Preparing Sample Data\n",
|
|
"\n",
|
|
"Let's create a dataset with some intentional duplicates. We'll simulate data coming from different sources (e.g., a CRM and a public database).\n",
|
|
"\n",
|
|
"**Our Entities:**\n",
|
|
"- **Apple**: Variations like \"Apple Inc.\", \"Apple Inc\", and just \"Apple\".\n",
|
|
"- **Microsoft**: Variations like \"Microsoft Corp\" and \"Microsoft\".\n",
|
|
"- **Google**: A unique entity for control."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"entities = [\n",
|
|
" # Apple Variations\n",
|
|
" {\n",
|
|
" \"id\": \"e1\",\n",
|
|
" \"name\": \"Apple Inc.\",\n",
|
|
" \"type\": \"Company\",\n",
|
|
" \"properties\": {\"industry\": \"Technology\", \"hq\": \"Cupertino\", \"founded\": 1976},\n",
|
|
" \"relationships\": [{\"predicate\": \"founded_by\", \"object\": \"Steve Jobs\"}]\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"e2\",\n",
|
|
" \"name\": \"Apple Inc\",\n",
|
|
" \"type\": \"Company\",\n",
|
|
" \"properties\": {\"industry\": \"Tech\", \"hq\": \"Cupertino, CA\"}, # Slightly different properties\n",
|
|
" \"relationships\": []\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"e3\",\n",
|
|
" \"name\": \"Apple\",\n",
|
|
" \"type\": \"Company\",\n",
|
|
" \"properties\": {\"industry\": \"Consumer Electronics\"}, \n",
|
|
" \"relationships\": [{\"predicate\": \"ceo\", \"object\": \"Tim Cook\"}]\n",
|
|
" },\n",
|
|
" \n",
|
|
" # Microsoft Variations\n",
|
|
" {\n",
|
|
" \"id\": \"e4\",\n",
|
|
" \"name\": \"Microsoft Corp\",\n",
|
|
" \"type\": \"Company\",\n",
|
|
" \"properties\": {\"industry\": \"Software\", \"hq\": \"Redmond\"}\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"e5\",\n",
|
|
" \"name\": \"Microsoft\",\n",
|
|
" \"type\": \"Company\",\n",
|
|
" \"properties\": {\"industry\": \"Tech\", \"hq\": \"Redmond, WA\"}\n",
|
|
" },\n",
|
|
" \n",
|
|
" # Unique Entity\n",
|
|
" {\n",
|
|
" \"id\": \"e6\",\n",
|
|
" \"name\": \"Google LLC\",\n",
|
|
" \"type\": \"Company\",\n",
|
|
" \"properties\": {\"industry\": \"Internet\"}\n",
|
|
" }\n",
|
|
"]\n",
|
|
"\n",
|
|
"print(f\"Created {len(entities)} sample entities.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Similarity Calculation\n",
|
|
"\n",
|
|
"The `SimilarityCalculator` is the core engine. It compares two entities and returns a score between 0 and 1. It looks at:\n",
|
|
"- **String Similarity**: Names and text fields.\n",
|
|
"- **Property Similarity**: Overlap in key-value pairs.\n",
|
|
"- **Relationship Similarity**: Connections to other entities.\n",
|
|
"- **Embeddings**: Semantic vector similarity (if available).\n",
|
|
"\n",
|
|
"You can customize the weights for each factor."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from semantica.deduplication import SimilarityCalculator, SimilarityResult\n",
|
|
"# Initialize calculator with custom weights\n",
|
|
"calculator = SimilarityCalculator(\n",
|
|
" string_weight=0.5, # High importance on name\n",
|
|
" property_weight=0.3, # Medium importance on properties\n",
|
|
" relationship_weight=0.2 # Lower importance on relationships\n",
|
|
")\n",
|
|
"\n",
|
|
"# Compare \"Apple Inc.\" (e1) vs \"Apple Inc\" (e2)\n",
|
|
"score_e1_e2 = calculator.calculate_similarity(entities[0], entities[1])\n",
|
|
"\n",
|
|
"print(f\"Similarity between '{entities[0]['name']}' and '{entities[1]['name']}':\")\n",
|
|
"print(f\" Total Score: {score_e1_e2.score:.4f}\")\n",
|
|
"print(f\" Breakdown: {score_e1_e2.components}\")\n",
|
|
"\n",
|
|
"# Compare \"Apple Inc.\" (e1) vs \"Microsoft\" (e5)\n",
|
|
"score_e1_e5 = calculator.calculate_similarity(entities[0], entities[4])\n",
|
|
"\n",
|
|
"print(f\"\\nSimilarity between '{entities[0]['name']}' and '{entities[4]['name']}':\")\n",
|
|
"print(f\" Total Score: {score_e1_e5.score:.4f}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Duplicate Detection\n",
|
|
"\n",
|
|
"The `DuplicateDetector` uses the similarity calculator to scan your dataset for duplicates. It can find:\n",
|
|
"- **Pairs**: Simple A matches B.\n",
|
|
"- **Groups**: A matches B, and B matches C.\n",
|
|
"\n",
|
|
"It uses a `similarity_threshold` to decide what counts as a match."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Import specific classes for Duplicate Detection\n",
|
|
"from semantica.deduplication import DuplicateDetector, DuplicateCandidate, DuplicateGroup\n",
|
|
"from semantica.deduplication import DeduplicationConfig\n",
|
|
"\n",
|
|
"detector = DuplicateDetector(\n",
|
|
" similarity_threshold=0.7, \n",
|
|
" confidence_threshold=0.6 \n",
|
|
")\n",
|
|
"\n",
|
|
"# Detect pairs\n",
|
|
"candidates = detector.detect_duplicates(entities)\n",
|
|
"\n",
|
|
"print(f\"Found {len(candidates)} duplicate pairs:\")\n",
|
|
"for c in candidates:\n",
|
|
" print(f\" - {c.entity1['name']} <==> {c.entity2['name']} (Score: {c.similarity_score:.2f})\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Incremental Detection\n",
|
|
"If you have an existing database and ingest new data, you don't want to re-compare everything. Use `incremental_detect`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"existing_db = entities[:3] # The Apple entities\n",
|
|
"new_data = [entities[4]] # Microsoft\n",
|
|
"# Check if new data matches anything in existing DB\n",
|
|
"inc_candidates = detector.incremental_detect(new_data, existing_db)\n",
|
|
"\n",
|
|
"print(f\"New matches found: {len(inc_candidates)}\")\n",
|
|
"# Expected: 0, because Microsoft is not Apple."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Clustering\n",
|
|
"\n",
|
|
"Sometimes pairs aren't enough. `ClusterBuilder` groups related entities into clusters. This is useful for understanding the full scope of a duplicated entity."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Import specific classes for Clustering\n",
|
|
"from semantica.deduplication import ClusterBuilder, Cluster, ClusterResult\n",
|
|
"cluster_builder = ClusterBuilder(threshold=0.7)\n",
|
|
"result = cluster_builder.build_clusters(entities)\n",
|
|
"\n",
|
|
"print(f\"Found {len(result.clusters)} clusters:\")\n",
|
|
"for i, cluster in enumerate(result.clusters):\n",
|
|
" names = [e['name'] for e in cluster.entities]\n",
|
|
" print(f\" Cluster {i+1}: {names}\")\n",
|
|
"\n",
|
|
"cluster_builder = ClusterBuilder(threshold=0.7)\n",
|
|
"result = cluster_builder.build_clusters(entities)\n",
|
|
"\n",
|
|
"print(f\"Found {len(result.clusters)} clusters:\")\n",
|
|
"for i, cluster in enumerate(result.clusters):\n",
|
|
" names = [e['name'] for e in cluster.entities]\n",
|
|
" print(f\" Cluster {i+1}: {names}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 5. Entity Merging\n",
|
|
"\n",
|
|
"Once duplicates are found, `EntityMerger` combines them. You need to choose a **Merge Strategy**:\n",
|
|
"\n",
|
|
"- `KEEP_FIRST` / `KEEP_LAST`: Based on order.\n",
|
|
"- `KEEP_MOST_COMPLETE`: Keeps the entity with the most data (properties + relationships).\n",
|
|
"- `KEEP_HIGHEST_CONFIDENCE`: Uses internal confidence scores.\n",
|
|
"- `MERGE_ALL`: Combines everything (arrays are concatenated, conflicts resolved by voting)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Import specific classes for Entity Merging\n",
|
|
"from semantica.deduplication import EntityMerger, MergeStrategy, MergeStrategyManager, MergeOperation, MergeResult"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"merger = EntityMerger()\n",
|
|
"\n",
|
|
"# We will use the 'KEEP_MOST_COMPLETE' strategy\n",
|
|
"# This ensures we don't lose valuable information from richer entities\n",
|
|
"merge_ops = merger.merge_duplicates(\n",
|
|
" entities, \n",
|
|
" strategy=MergeStrategy.KEEP_MOST_COMPLETE\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"Performed {len(merge_ops)} merge operations.\")\n",
|
|
"\n",
|
|
"print(\"\\n--- Merged Results ---\")\n",
|
|
"for op in merge_ops:\n",
|
|
" final_ent = op.merged_entity\n",
|
|
" original_count = len(op.source_entities)\n",
|
|
" print(f\"Merged {original_count} entities into: '{final_ent['name']}'\")\n",
|
|
" print(f\" - Final Properties: {final_ent['properties']}\")\n",
|
|
" print(f\" - Final Relationships: {len(final_ent.get('relationships', []))}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 6. Complete Workflow\n",
|
|
"\n",
|
|
"Let's wrap this up into a clean function that takes dirty data and returns clean data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def deduplicate_dataset(raw_entities):\n",
|
|
" print(\"1. Detecting duplicates...\")\n",
|
|
" # Step 1: Detect\n",
|
|
" detector = DuplicateDetector(similarity_threshold=0.75)\n",
|
|
" # We can skip explicit detection calls if we just want to merge, \n",
|
|
" # as EntityMerger calls detection internally, but doing it manually allows inspection.\n",
|
|
" \n",
|
|
" print(\"2. Merging entities...\")\n",
|
|
" # Step 2: Merge\n",
|
|
" merger = EntityMerger()\n",
|
|
" ops = merger.merge_duplicates(raw_entities, strategy=MergeStrategy.KEEP_MOST_COMPLETE)\n",
|
|
" \n",
|
|
" # Let's collect all final IDs to see what remains\n",
|
|
" merged_entities = [op.merged_entity for op in ops]\n",
|
|
" \n",
|
|
" # Find entities that were NOT part of any merge (singletons)\n",
|
|
" merged_ids = set()\n",
|
|
" for op in ops:\n",
|
|
" for source in op.source_entities:\n",
|
|
" merged_ids.add(source['id'])\n",
|
|
" \n",
|
|
" singletons = [e for e in raw_entities if e['id'] not in merged_ids]\n",
|
|
" \n",
|
|
" final_dataset = merged_entities + singletons\n",
|
|
" return final_dataset\n",
|
|
"\n",
|
|
"# Run the workflow\n",
|
|
"clean_data = deduplicate_dataset(entities)\n",
|
|
"\n",
|
|
"print(f\"\\nOriginal Size: {len(entities)}\")\n",
|
|
"print(f\"Cleaned Size: {len(clean_data)}\")\n",
|
|
"print(\"\\nFinal Entity Names:\")\n",
|
|
"for e in clean_data:\n",
|
|
" print(f\" - {e['name']}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Summary\n",
|
|
"\n",
|
|
"You've learned how to:\n",
|
|
"1. **Import** the necessary Deduplication classes.\n",
|
|
"2. **Calculate Similarity** between entities.\n",
|
|
"3. **Detect Duplicates** using configurable thresholds.\n",
|
|
"4. **Cluster** similar entities.\n",
|
|
"5. **Merge** duplicates into a clean, canonical dataset.\n",
|
|
"\n",
|
|
"This module is essential for maintaining high-quality Knowledge Graphs, especially when ingesting data from multiple, potentially messy sources."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|