|
|
|
@@ -4,23 +4,38 @@
|
|
|
|
|
"cell_type": "markdown",
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"source": [
|
|
|
|
|
"# Conflict Detection and Resolution\n",
|
|
|
|
|
"# Conflict Detection and Resolution in Semantica\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"## Overview\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"In real-world Knowledge Graph construction, data often comes from multiple heterogeneous sources (databases, APIs, files, streams). These sources may provide conflicting information about the same entities or relationships. \n",
|
|
|
|
|
"In modern data pipelines, especially those building Knowledge Graphs, data is often ingested from multiple heterogeneous sources (e.g., internal databases, third-party APIs, web scrapes). Discrepancies are inevitable. \n",
|
|
|
|
|
"\n",
|
|
|
|
|
"The **Semantica Conflict Detection and Resolution** module (`semantica.conflicts`) provides a comprehensive suite of tools to identifying, analyzing, and resolving these discrepancies to ensure high data quality and trust.\n",
|
|
|
|
|
"The **Semantica Conflict Resolution Module** (`semantica.conflicts`) provides a robust framework for managing these data inconsistencies. It is designed to ensure that your downstream applications consume only high-quality, reconciled data.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"**Key Capabilities:**\n",
|
|
|
|
|
"- **Conflict Detection**: Identify value mismatches, type inconsistencies, and temporal contradictions.\n",
|
|
|
|
|
"- **Source Tracking**: Trace every piece of data back to its origin with credibility scoring.\n",
|
|
|
|
|
"- **Resolution Strategies**: Apply automated strategies like voting, credibility weighting, or recency.\n",
|
|
|
|
|
"- **Investigation Guides**: Generate human-readable guides for complex conflicts requiring manual review.\n",
|
|
|
|
|
"### Key Capabilities\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"1. **Multi-Dimensional Conflict Detection**\n",
|
|
|
|
|
" * **Value Conflicts**: Different values for the same property (e.g., `\"Google\"` vs `\"Google Inc.\"`).\n",
|
|
|
|
|
" * **Type Conflicts**: Data type mismatches (e.g., string vs integer).\n",
|
|
|
|
|
" * **Temporal Conflicts**: Chronological inconsistencies (e.g., a `start_date` after an `end_date`).\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"2. **Provenance & Source Tracking**\n",
|
|
|
|
|
" * **Granular Tracking**: Trace every property value back to its specific source document, page, or API call.\n",
|
|
|
|
|
" * **Credibility Scoring**: Assign trust scores to sources (e.g., `0.95` for internal HR DB vs `0.60` for web scrapes).\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"3. **Automated Resolution Strategies**\n",
|
|
|
|
|
" * **Voting**: Majority rules (useful for multiple equal-weight sources).\n",
|
|
|
|
|
" * **Credibility Weighted**: Values from higher-trust sources override others.\n",
|
|
|
|
|
" * **Recency**: The most recent data point wins.\n",
|
|
|
|
|
" * **Expert Review**: Flag complex conflicts for human intervention.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"4. **Investigation & Auditing**\n",
|
|
|
|
|
" * **Investigation Guides**: Auto-generate step-by-step guides for human analysts to resolve sticky conflicts.\n",
|
|
|
|
|
" * **Audit Trails**: Keep a record of how every conflict was resolved for compliance.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"## Installation\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"Ensure Semantica is installed:\n",
|
|
|
|
|
"Ensure Semantica is installed in your environment:\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"```bash\n",
|
|
|
|
|
"pip install semantica\n",
|
|
|
|
@@ -51,7 +66,17 @@
|
|
|
|
|
"source": [
|
|
|
|
|
"## Step 1: Simulating Multi-Source Data\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"Let's simulate a scenario where we receive data about the same person from three different sources: an HR database, a LinkedIn scrape, and a public directory. Note the discrepancies in `birth_date` and `department`."
|
|
|
|
|
"To demonstrate the framework, we will simulate a realistic scenario involving employee data.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"**The Scenario:**\n",
|
|
|
|
|
"We have received records for **Employee 001** from three distinct sources:\n",
|
|
|
|
|
"1. **HR Database**: Highly trusted internal source.\n",
|
|
|
|
|
"2. **LinkedIn Scrape**: Less reliable external source.\n",
|
|
|
|
|
"3. **Public Directory**: Outdated public API.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"**The Conflicts:**\n",
|
|
|
|
|
"* **`birth_date`**: The Public Directory lists a different year.\n",
|
|
|
|
|
"* **`department`**: LinkedIn uses a more specific name (\"Software Engineering\") vs the generic \"Engineering\" in the HR DB."
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
@@ -68,14 +93,14 @@
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"source": [
|
|
|
|
|
"# Define simulated data sources\n",
|
|
|
|
|
"sources = {\n",
|
|
|
|
|
"# 1. Define source metadata\n",
|
|
|
|
|
"sources_metadata = {\n",
|
|
|
|
|
" \"hr_db\": {\"credibility\": 0.95, \"type\": \"internal_database\"},\n",
|
|
|
|
|
" \"linkedin_scrape\": {\"credibility\": 0.60, \"type\": \"web_scrape\"},\n",
|
|
|
|
|
" \"public_dir\": {\"credibility\": 0.40, \"type\": \"public_api\"}\n",
|
|
|
|
|
"}\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# Define entities from these sources\n",
|
|
|
|
|
"# 2. Define entity records from these sources\n",
|
|
|
|
|
"entity_records = [\n",
|
|
|
|
|
" {\n",
|
|
|
|
|
" \"id\": \"emp_001\",\n",
|
|
|
|
@@ -96,7 +121,7 @@
|
|
|
|
|
" {\n",
|
|
|
|
|
" \"id\": \"emp_001\",\n",
|
|
|
|
|
" \"name\": \"John Doe\",\n",
|
|
|
|
|
" \"birth_date\": \"1982-05-15\", # Conflict!\n",
|
|
|
|
|
" \"birth_date\": \"1982-05-15\", # Conflict: Different year\n",
|
|
|
|
|
" \"department\": \"Engineering\",\n",
|
|
|
|
|
" \"source\": \"public_dir\",\n",
|
|
|
|
|
" \"timestamp\": \"2022-12-01T09:00:00\"\n",
|
|
|
|
@@ -110,40 +135,33 @@
|
|
|
|
|
"cell_type": "markdown",
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"source": [
|
|
|
|
|
"## Step 2: Tracking Sources\n",
|
|
|
|
|
"## Step 2: Registering and Tracking Sources\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"Before detecting conflicts, we register our sources with the `SourceTracker`. This allows the system to factor in source credibility during resolution."
|
|
|
|
|
"Before we can effectively resolve conflicts based on trust, we must register our sources with the `SourceTracker`.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"The `SourceTracker` acts as a central registry for:\n",
|
|
|
|
|
"* **Credibility Scores**: How much we trust the source.\n",
|
|
|
|
|
"* **Metadata**: Source type, location, and update frequency.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"We iterate through our simulated sources and register them."
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"cell_type": "code",
|
|
|
|
|
"execution_count": 8,
|
|
|
|
|
"execution_count": 3,
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"outputs": [
|
|
|
|
|
{
|
|
|
|
|
"ename": "AttributeError",
|
|
|
|
|
"evalue": "'SourceTracker' object has no attribute 'register_source'",
|
|
|
|
|
"output_type": "error",
|
|
|
|
|
"traceback": [
|
|
|
|
|
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
|
|
|
|
"\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)",
|
|
|
|
|
"Cell \u001b[1;32mIn[8], line 7\u001b[0m\n\u001b[0;32m 4\u001b[0m source_tracker \u001b[38;5;241m=\u001b[39m SourceTracker()\n\u001b[0;32m 6\u001b[0m \u001b[38;5;66;03m# 4. Now this will work\u001b[39;00m\n\u001b[1;32m----> 7\u001b[0m \u001b[43msource_tracker\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mregister_source\u001b[49m(\n\u001b[0;32m 8\u001b[0m source_id\u001b[38;5;241m=\u001b[39msource_id,\n\u001b[0;32m 9\u001b[0m source_type\u001b[38;5;241m=\u001b[39mmetadata[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtype\u001b[39m\u001b[38;5;124m\"\u001b[39m],\n\u001b[0;32m 10\u001b[0m credibility_score\u001b[38;5;241m=\u001b[39mmetadata[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcredibility\u001b[39m\u001b[38;5;124m\"\u001b[39m]\n\u001b[0;32m 11\u001b[0m )\n",
|
|
|
|
|
"\u001b[1;31mAttributeError\u001b[0m: 'SourceTracker' object has no attribute 'register_source'"
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"outputs": [],
|
|
|
|
|
"source": [
|
|
|
|
|
"from semantica.conflicts import SourceTracker\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# 3. Create a new instance\n",
|
|
|
|
|
"source_tracker = SourceTracker()\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# 4. Now this will work\n",
|
|
|
|
|
"source_tracker.register_source(\n",
|
|
|
|
|
" source_id=source_id,\n",
|
|
|
|
|
" source_type=metadata[\"type\"],\n",
|
|
|
|
|
" credibility_score=metadata[\"credibility\"]\n",
|
|
|
|
|
")"
|
|
|
|
|
"print(\"Registering sources...\")\n",
|
|
|
|
|
"for source_id, metadata in sources_metadata.items():\n",
|
|
|
|
|
" source_tracker.register_source(\n",
|
|
|
|
|
" source_id=source_id,\n",
|
|
|
|
|
" source_type=metadata[\"type\"],\n",
|
|
|
|
|
" credibility_score=metadata[\"credibility\"]\n",
|
|
|
|
|
" )\n",
|
|
|
|
|
" print(f\" - Registered '{source_id}' with credibility {metadata['credibility']}\")"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
@@ -152,36 +170,20 @@
|
|
|
|
|
"source": [
|
|
|
|
|
"## Step 3: Detecting Conflicts\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"Now we use `ConflictDetector` to identify discrepancies. We'll check for value conflicts in `birth_date` and `department`.\n",
|
|
|
|
|
"We use the `ConflictDetector` to scan our records for discrepancies. \n",
|
|
|
|
|
"\n",
|
|
|
|
|
"The detector compares values across all records for the same entity ID."
|
|
|
|
|
"The detector is flexible and can be configured to check:\n",
|
|
|
|
|
"* **Specific Properties**: Check only critical fields like `birth_date`.\n",
|
|
|
|
|
"* **Entire Entities**: Scan all properties for an entity.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"Here, we explicitly check `birth_date` and `department`."
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"cell_type": "code",
|
|
|
|
|
"execution_count": 7,
|
|
|
|
|
"execution_count": 4,
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"outputs": [
|
|
|
|
|
{
|
|
|
|
|
"data": {
|
|
|
|
|
"text/html": [
|
|
|
|
|
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>File</th><th>Time</th></tr><tr><td>✅</td><td>Semantica is resolving</td><td>⚠️ conflicts</td><td>ConflictDetector</td><td>-</td><td>0.01s</td></tr><tr><td>✅</td><td>Semantica is resolving</td><td>⚠️ conflicts</td><td>ConflictAnalyzer</td><td>-</td><td>0.01s</td></tr><tr><td>✅</td><td>Semantica is resolving</td><td>⚠️ conflicts</td><td>InvestigationGuideGenerator</td><td>-</td><td>0.00s</td></tr></table></div>"
|
|
|
|
|
],
|
|
|
|
|
"text/plain": [
|
|
|
|
|
"<IPython.core.display.HTML object>"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"output_type": "display_data"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "stderr",
|
|
|
|
|
"output_type": "stream",
|
|
|
|
|
"text": [
|
|
|
|
|
"Value conflict detected: emp_001.birth_date has conflicting values: ['1980-05-15', '1982-05-15']\n",
|
|
|
|
|
"Value conflict detected: emp_001.department has conflicting values: ['Software Engineering', 'Engineering']\n"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "stdout",
|
|
|
|
|
"output_type": "stream",
|
|
|
|
@@ -199,16 +201,16 @@
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"source": [
|
|
|
|
|
"detector = ConflictDetector()\n",
|
|
|
|
|
"# Initialize detector with our populated source tracker\n",
|
|
|
|
|
"detector = ConflictDetector(source_tracker=source_tracker)\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# Detect conflicts for specific properties\n",
|
|
|
|
|
"conflicts = []\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# Check birth_date\n",
|
|
|
|
|
"# 1. Check birth_date\n",
|
|
|
|
|
"dob_conflicts = detector.detect_value_conflicts(entity_records, \"birth_date\")\n",
|
|
|
|
|
"conflicts.extend(dob_conflicts)\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# Check department\n",
|
|
|
|
|
"# 2. Check department\n",
|
|
|
|
|
"dept_conflicts = detector.detect_value_conflicts(entity_records, \"department\")\n",
|
|
|
|
|
"conflicts.extend(dept_conflicts)\n",
|
|
|
|
|
"\n",
|
|
|
|
@@ -224,14 +226,17 @@
|
|
|
|
|
"cell_type": "markdown",
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"source": [
|
|
|
|
|
"## Step 4: Analyzing Patterns\n",
|
|
|
|
|
"## Step 4: Analyzing Conflict Patterns\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"The `ConflictAnalyzer` can help identify systemic issues, such as a specific source consistently contradicting others."
|
|
|
|
|
"When dealing with large datasets, individual conflicts are less important than systemic patterns. The `ConflictAnalyzer` helps answer questions like:\n",
|
|
|
|
|
"* \"Is one specific source responsible for most conflicts?\"\n",
|
|
|
|
|
"* \"Are conflicts concentrated in a specific entity type?\"\n",
|
|
|
|
|
"* \"What is the distribution of conflict severity?\""
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"cell_type": "code",
|
|
|
|
|
"execution_count": 9,
|
|
|
|
|
"execution_count": 5,
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"outputs": [
|
|
|
|
|
{
|
|
|
|
@@ -251,8 +256,8 @@
|
|
|
|
|
"\n",
|
|
|
|
|
"print(\"Conflict Analysis Summary:\")\n",
|
|
|
|
|
"print(f\"Total Conflicts: {analysis['total_conflicts']}\")\n",
|
|
|
|
|
"print(f\"By Type: {analysis['by_type']}\")\n",
|
|
|
|
|
"print(f\"By Severity: {analysis['by_severity']}\")"
|
|
|
|
|
"print(f\"By Type: {analysis.get('by_type', {}).get('counts')}\")\n",
|
|
|
|
|
"print(f\"By Severity: {analysis.get('by_severity', {}).get('counts')}\")"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
@@ -261,52 +266,43 @@
|
|
|
|
|
"source": [
|
|
|
|
|
"## Step 5: Resolving Conflicts\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"We can resolve conflicts using different strategies. \n",
|
|
|
|
|
"This is the critical step where we decide which value to trust. Semantica offers flexible `ResolutionStrategies`.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"### Strategy A: Voting\n",
|
|
|
|
|
"Uses the most frequent value. Useful when you have many sources of equal standing.\n",
|
|
|
|
|
"### Strategy A: Voting (Majority Rules)\n",
|
|
|
|
|
"This strategy selects the value that appears most frequently. It is simple but treats all sources as equal.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"### Strategy B: Credibility Weighted\n",
|
|
|
|
|
"Prefers values from trusted sources (like our HR DB) over lower-trust sources (public directory)."
|
|
|
|
|
"This strategy calculates a weighted score for each value based on the `credibility` of its source. \n",
|
|
|
|
|
"\n",
|
|
|
|
|
"**Example:**\n",
|
|
|
|
|
"* `hr_db` (0.95) says \"1980-05-15\"\n",
|
|
|
|
|
"* `public_dir` (0.40) says \"1982-05-15\"\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"Even if multiple low-quality sources agreed on the wrong date, the high-credibility source would likely win."
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"cell_type": "code",
|
|
|
|
|
"execution_count": 10,
|
|
|
|
|
"execution_count": 6,
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"outputs": [
|
|
|
|
|
{
|
|
|
|
|
"ename": "AttributeError",
|
|
|
|
|
"evalue": "'ConflictResolver' object has no attribute 'set_source_tracker'",
|
|
|
|
|
"output_type": "error",
|
|
|
|
|
"traceback": [
|
|
|
|
|
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
|
|
|
|
"\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)",
|
|
|
|
|
"Cell \u001b[1;32mIn[10], line 4\u001b[0m\n\u001b[0;32m 1\u001b[0m resolver \u001b[38;5;241m=\u001b[39m ConflictResolver()\n\u001b[0;32m 3\u001b[0m \u001b[38;5;66;03m# Need to link the source tracker to the resolver for credibility strategies\u001b[39;00m\n\u001b[1;32m----> 4\u001b[0m \u001b[43mresolver\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mset_source_tracker\u001b[49m(source_tracker)\n\u001b[0;32m 6\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m--- Resolution: Voting ---\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m 7\u001b[0m voting_results \u001b[38;5;241m=\u001b[39m resolver\u001b[38;5;241m.\u001b[39mresolve_conflicts(conflicts, strategy\u001b[38;5;241m=\u001b[39mResolutionStrategy\u001b[38;5;241m.\u001b[39mVOTING)\n",
|
|
|
|
|
"\u001b[1;31mAttributeError\u001b[0m: 'ConflictResolver' object has no attribute 'set_source_tracker'"
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"outputs": [],
|
|
|
|
|
"source": [
|
|
|
|
|
"resolver = ConflictResolver()\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# Need to link the source tracker to the resolver for credibility strategies\n",
|
|
|
|
|
"# CRITICAL: Link the source tracker to the resolver.\n",
|
|
|
|
|
"# This allows the resolver to look up the credibility scores we registered in Step 2.\n",
|
|
|
|
|
"resolver.set_source_tracker(source_tracker)\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"print(\"--- Resolution: Voting ---\")\n",
|
|
|
|
|
"voting_results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING)\n",
|
|
|
|
|
"for res in voting_results:\n",
|
|
|
|
|
" print(f\"Property: {res.metadata.get('property_name')}\")\n",
|
|
|
|
|
" print(f\"Resolved Value: {res.resolved_value}\")\n",
|
|
|
|
|
" print(f\"Confidence: {res.confidence:.2f}\")\n",
|
|
|
|
|
" print(f\"Property: {res.metadata.get('property_name'):<15} | Resolved Value: {res.resolved_value}\")\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"print(\"\\n--- Resolution: Credibility Weighted ---\")\n",
|
|
|
|
|
"# This should favor the HR DB value for birth_date\n",
|
|
|
|
|
"# Notice how the HR DB's value is preferred due to higher credibility\n",
|
|
|
|
|
"credibility_results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED)\n",
|
|
|
|
|
"for res in credibility_results:\n",
|
|
|
|
|
" print(f\"Property: {res.metadata.get('property_name')}\")\n",
|
|
|
|
|
" print(f\"Resolved Value: {res.resolved_value}\")\n",
|
|
|
|
|
" print(f\"Confidence: {res.confidence:.2f}\")"
|
|
|
|
|
" print(f\"Property: {res.metadata.get('property_name'):<15} | Resolved Value: {res.resolved_value} (Confidence: {res.confidence:.2f})\")"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
@@ -315,48 +311,36 @@
|
|
|
|
|
"source": [
|
|
|
|
|
"## Step 6: Generating Investigation Guides\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"For critical conflicts or those with low resolution confidence, manual intervention is needed. The `InvestigationGuideGenerator` creates a structured guide for human analysts."
|
|
|
|
|
"Not all conflicts can be resolved automatically. High-stakes or low-confidence resolutions require human review.\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"The `InvestigationGuideGenerator` produces a structured \"flight plan\" for an analyst, detailing:\n",
|
|
|
|
|
"1. **What** is in conflict.\n",
|
|
|
|
|
"2. **Who** (which sources) are involved.\n",
|
|
|
|
|
"3. **How** to verify the correct data (actionable steps)."
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"cell_type": "code",
|
|
|
|
|
"execution_count": 11,
|
|
|
|
|
"execution_count": 7,
|
|
|
|
|
"metadata": {},
|
|
|
|
|
"outputs": [
|
|
|
|
|
{
|
|
|
|
|
"name": "stdout",
|
|
|
|
|
"output_type": "stream",
|
|
|
|
|
"text": [
|
|
|
|
|
"Investigation Guide for emp_001_birth_date_conflict:\n"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"ename": "AttributeError",
|
|
|
|
|
"evalue": "'InvestigationGuide' object has no attribute 'title'",
|
|
|
|
|
"output_type": "error",
|
|
|
|
|
"traceback": [
|
|
|
|
|
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
|
|
|
|
"\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)",
|
|
|
|
|
"Cell \u001b[1;32mIn[11], line 7\u001b[0m\n\u001b[0;32m 4\u001b[0m guide \u001b[38;5;241m=\u001b[39m guide_generator\u001b[38;5;241m.\u001b[39mgenerate_guide(conflicts[\u001b[38;5;241m0\u001b[39m])\n\u001b[0;32m 6\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mInvestigation Guide for \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mguide\u001b[38;5;241m.\u001b[39mconflict_id\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m:\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m----> 7\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTitle: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[43mguide\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtitle\u001b[49m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m 8\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSteps:\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m 9\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i, step \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(guide\u001b[38;5;241m.\u001b[39msteps, \u001b[38;5;241m1\u001b[39m):\n",
|
|
|
|
|
"\u001b[1;31mAttributeError\u001b[0m: 'InvestigationGuide' object has no attribute 'title'"
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"outputs": [],
|
|
|
|
|
"source": [
|
|
|
|
|
"guide_generator = InvestigationGuideGenerator()\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"# Generate a guide for the first conflict (e.g., birth_date)\n",
|
|
|
|
|
"# Generate a guide for the first conflict (birth_date)\n",
|
|
|
|
|
"guide = guide_generator.generate_guide(conflicts[0])\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"print(f\"Investigation Guide for {guide.conflict_id}:\")\n",
|
|
|
|
|
"print(f\"Title: {guide.title}\")\n",
|
|
|
|
|
"print(\"Steps:\")\n",
|
|
|
|
|
"for i, step in enumerate(guide.steps, 1):\n",
|
|
|
|
|
" print(f\"{i}. {step.description} (Action: {step.action_type})\")\n",
|
|
|
|
|
"print(f\"=== {guide.title} ===\")\n",
|
|
|
|
|
"print(f\"Summary: {guide.conflict_summary}\\n\")\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"print(\"\\nRecommended Checks:\")\n",
|
|
|
|
|
"for check in guide.checklist:\n",
|
|
|
|
|
" print(f\"[ ] {check}\")"
|
|
|
|
|
"print(\"Investigation Steps:\")\n",
|
|
|
|
|
"for i, step in enumerate(guide.investigation_steps, 1):\n",
|
|
|
|
|
" print(f\"{i}. {step.description}\")\n",
|
|
|
|
|
" print(f" Action: {step.action}\")\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"print(\"\\nRecommended Actions:\")\n",
|
|
|
|
|
"for action in guide.recommended_actions:\n",
|
|
|
|
|
" print(f"[ ] {action}\")"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
@@ -365,13 +349,14 @@
|
|
|
|
|
"source": [
|
|
|
|
|
"## Conclusion\n",
|
|
|
|
|
"\n",
|
|
|
|
|
"In this notebook, we explored how to:\n",
|
|
|
|
|
"1. **Detect** conflicts in multi-source data.\n",
|
|
|
|
|
"2. **Track** data provenance and source credibility.\n",
|
|
|
|
|
"3. **Resolve** conflicts using automated strategies tailored to your data governance needs.\n",
|
|
|
|
|
"4. **Investigate** complex issues with generated guides.\n",
|
|
|
|
|
"You have successfully built a conflict resolution pipeline using Semantica! \n",
|
|
|
|
|
"\n",
|
|
|
|
|
"By integrating these steps into your pipeline, you ensure your Knowledge Graph remains accurate, consistent, and trustworthy."
|
|
|
|
|
"**Recap of what we achieved:**\n",
|
|
|
|
|
"1. **Ingested** multi-source data with potential quality issues.\n",
|
|
|
|
|
"2. **Registered** sources with credibility scores to establish a \"hierarchy of trust.\"\n",
|
|
|
|
|
"3. **Detected** conflicts automatically across the dataset.\n",
|
|
|
|
|
"4. **Resolved** conflicts using a credibility-weighted strategy, prioritizing trusted internal data.\n",
|
|
|
|
|
"5. **Generated** actionable guides for human-in-the-loop review of difficult cases."
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
KaifAhmad1