{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conflict Detection and Resolution\n",
    "\n",
    "## Overview\n",
    "\n",
    "In modern data pipelines, especially those building Knowledge Graphs, data is often ingested from multiple heterogeneous sources (e.g., internal databases, third-party APIs, web scrapes). Discrepancies are inevitable. \n",
    "\n",
    "The **Semantica Conflict Resolution Module** (`semantica.conflicts`) provides a robust framework for managing these data inconsistencies. It is designed to ensure that your downstream applications consume only high-quality, reconciled data.\n",
    "\n",
    "### What counts as a \"conflict\"?\n",
    "\n",
    "- A conflict happens when **multiple records for the same entity** disagree on a field.\n",
    "- Semantica typically assumes each record is a dictionary with:\n",
    "  - `id` (or `entity_id`): stable identifier for the entity being described\n",
    "  - one or more attributes (e.g., `name`, `birth_date`, `department`)\n",
    "  - `source`: where the value came from (db, scrape, api, file)\n",
    "  - optional `timestamp`: when the value was observed\n",
    "- The module is source-aware: it can record **which sources contributed which values**, then resolve using strategies like voting or credibility.\n",
    "\n",
    "### Practical API notes (to avoid common mismatches)\n",
    "\n",
    "- Use `ConflictDetector.detect_value_conflicts(entities, property_name=...)` when you want to check one field.\n",
    "- Use `ConflictDetector.detect_conflicts(entities)` when you want a broader scan (value/type/temporal/etc.).\n",
    "- If you're starting from a Knowledge Graph dictionary (built via `GraphBuilder`), pull entities via `kg.get(\"entities\", [])`.\n",
    "\n",
    "### Key Capabilities\n",
    "\n",
    "1.  **Multi-Dimensional Conflict Detection**\n",
    "    *   **Value Conflicts**: Different values for the same property (e.g., `\"Google\"` vs `\"Google Inc.\"`).\n",
    "    *   **Type Conflicts**: Data type mismatches (e.g., string vs integer).\n",
    "    *   **Temporal Conflicts**: Chronological inconsistencies (e.g., a `start_date` after an `end_date`).\n",
    "\n",
    "2.  **Provenance & Source Tracking**\n",
    "    *   **Granular Tracking**: Trace every property value back to its specific source document, page, or API call.\n",
    "    *   **Credibility Scoring**: Assign trust scores to sources (e.g., `0.95` for internal HR DB vs `0.60` for web scrapes).\n",
    "\n",
    "3.  **Automated Resolution Strategies**\n",
    "    *   **Voting**: Majority rules (useful for multiple equal-weight sources).\n",
    "    *   **Credibility Weighted**: Values from higher-trust sources override others.\n",
    "    *   **Recency**: The most recent data point wins.\n",
    "    *   **Expert Review**: Flag complex conflicts for human intervention.\n",
    "\n",
    "4.  **Investigation & Auditing**\n",
    "    *   **Investigation Guides**: Auto-generate step-by-step guides for human analysts to resolve sticky conflicts.\n",
    "    *   **Audit Trails**: Keep a record of how every conflict was resolved for compliance.\n",
    "\n",
    "## Installation\n",
    "\n",
    "Ensure Semantica is installed in your environment:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "```\n",
    "\n",
    "- If you're running this notebook inside the Semantica repo, prefer an editable install (so changes in code are reflected immediately):\n",
    "  - `pip install -e .`\n",
    "- If you're using a hosted notebook environment, `%pip install semantica` is often more reliable than `!pip install ...` because it installs into the active kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q semantica "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Simulating Multi-Source Data\n",
    "\n",
    "To demonstrate the framework, we will simulate a realistic scenario involving employee data.\n",
    "\n",
    "**The Scenario:**\n",
    "We have received records for **Employee 001** from three distinct sources:\n",
    "\n",
    "- **HR Database**: Highly trusted internal source.\n",
    "- **LinkedIn Scrape**: Less reliable external source.\n",
    "- **Public Directory**: Outdated public API.\n",
    "\n",
    "**The record shape (what Semantica expects):**\n",
    "\n",
    "- Each record is a dictionary describing the same entity (`id`: `emp_001`).\n",
    "- Each record includes a `source` key so conflicts can be attributed.\n",
    "- A `timestamp` lets you apply time-based resolution strategies (e.g., most recent wins).\n",
    "\n",
    "**The Conflicts:**\n",
    "*   **`birth_date`**: The Public Directory lists a different year.\n",
    "*   **`department`**: LinkedIn uses a more specific name (\"Software Engineering\") vs the generic \"Engineering\" in the HR DB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Define source metadata\n",
    "sources_metadata = {\n",
    "    \"hr_db\": {\"credibility\": 0.95, \"type\": \"internal_database\"},\n",
    "    \"linkedin_scrape\": {\"credibility\": 0.60, \"type\": \"web_scrape\"},\n",
    "    \"public_dir\": {\"credibility\": 0.40, \"type\": \"public_api\"}\n",
    "}\n",
    "\n",
    "# 2. Define entity records from these sources\n",
    "entity_records = [\n",
    "    {\n",
    "        \"id\": \"emp_001\",\n",
    "        \"name\": \"John Doe\",\n",
    "        \"birth_date\": \"1980-05-15\",\n",
    "        \"department\": \"Engineering\",\n",
    "        \"source\": \"hr_db\",\n",
    "        \"timestamp\": \"2023-01-01T10:00:00\"\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"emp_001\",\n",
    "        \"name\": \"Jonathan Doe\",\n",
    "        \"birth_date\": \"1980-05-15\",\n",
    "        \"department\": \"Software Engineering\",\n",
    "        \"source\": \"linkedin_scrape\",\n",
    "        \"timestamp\": \"2023-06-15T14:30:00\"\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"emp_001\",\n",
    "        \"name\": \"John Doe\",\n",
    "        \"birth_date\": \"1982-05-15\",  # Conflict: Different year\n",
    "        \"department\": \"Engineering\",\n",
    "        \"source\": \"public_dir\",\n",
    "        \"timestamp\": \"2022-12-01T09:00:00\"\n",
    "    }\n",
    "]\n",
    "\n",
    "print(f\"Loaded {len(entity_records)} records for Employee 001\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Registering and Tracking Sources\n",
    "\n",
    "Before we can effectively resolve conflicts based on trust, we must register our sources with the `SourceTracker`.\n",
    "\n",
    "The `SourceTracker` acts as a central registry for:\n",
    "\n",
    "- **Credibility Scores**: How much you trust the source.\n",
    "- **Metadata**: Helpful context (e.g., source type, system of record vs scrape).\n",
    "\n",
    "**How to think about credibility scores:**\n",
    "\n",
    "- Use scores as a *relative ordering* (the exact decimals matter less than the ranking).\n",
    "- Start simple: `internal_db > vendor_api > web_scrape`.\n",
    "- Revisit scores later using analytics (e.g., \"which sources are frequently wrong?\").\n",
    "\n",
    "We iterate through our simulated sources and register them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import SourceTracker\n",
    "\n",
    "source_tracker = SourceTracker()\n",
    "\n",
    "print(\"Registering sources...\")\n",
    "for source_id, metadata in sources_metadata.items():\n",
    "    source_tracker.register_source(\n",
    "        source_id=source_id,\n",
    "        source_type=metadata[\"type\"],\n",
    "        credibility_score=metadata[\"credibility\"]\n",
    "    )\n",
    "    print(f\"  - Registered '{source_id}' with credibility {metadata['credibility']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Detecting Conflicts\n",
    "\n",
    "We use the `ConflictDetector` to scan our records for discrepancies. \n",
    "\n",
    "The detector is flexible and can be configured to check:\n",
    "\n",
    "- **Specific properties**: Check only critical fields like `birth_date`.\n",
    "- **Entity-wide scans**: Scan many properties (or all) for an entity.\n",
    "\n",
    "**What you get back:**\n",
    "\n",
    "- A list of `Conflict` objects.\n",
    "- Useful fields you’ll typically inspect:\n",
    "  - `conflict_type` (e.g., `value_conflict`)\n",
    "  - `entity_id`, `property_name`\n",
    "  - `conflicting_values` and `sources`\n",
    "  - `severity` and `confidence`\n",
    "\n",
    "Here, we explicitly check `birth_date` and `department`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector\n",
    "\n",
    "# Initialize detector with our populated source tracker\n",
    "detector = ConflictDetector(source_tracker=source_tracker)\n",
    "\n",
    "conflicts = []\n",
    "\n",
    "# 1. Check birth_date\n",
    "dob_conflicts = detector.detect_value_conflicts(entity_records, \"birth_date\")\n",
    "conflicts.extend(dob_conflicts)\n",
    "\n",
    "# 2. Check department\n",
    "dept_conflicts = detector.detect_value_conflicts(entity_records, \"department\")\n",
    "conflicts.extend(dept_conflicts)\n",
    "\n",
    "print(f\"Detected {len(conflicts)} conflicts:\")\n",
    "for conflict in conflicts:\n",
    "    print(f\"- {conflict.conflict_type.value}: {conflict.property_name} for {conflict.entity_id}\")\n",
    "    print(f\"  Values: {conflict.conflicting_values}\")\n",
    "    print(f\"  Severity: {conflict.severity}\")\n",
    "    print(\"--- \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Analyzing Conflict Patterns\n",
    "\n",
    "When dealing with large datasets, individual conflicts are less important than systemic patterns. The `ConflictAnalyzer` helps answer questions like:\n",
    "\n",
    "- \"Is one specific source responsible for most conflicts?\"\n",
    "- \"Are conflicts concentrated in a specific entity type or property?\"\n",
    "- \"What is the distribution of conflict severity and conflict types?\"\n",
    "\n",
    "**How to use this in a pipeline:**\n",
    "\n",
    "- Run analysis to identify noisy sources.\n",
    "- Use results to adjust credibility scores (Step 2) or refine ingestion/cleaning rules.\n",
    "- Track trends over time to catch regressions in upstream systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictAnalyzer\n",
    "\n",
    "analyzer = ConflictAnalyzer()\n",
    "analysis = analyzer.analyze_conflicts(conflicts)\n",
    "\n",
    "print(\"Conflict Analysis Summary:\")\n",
    "print(f\"Total Conflicts: {analysis['total_conflicts']}\")\n",
    "print(f\"By Type: {analysis.get('by_type', {}).get('counts')}\")\n",
    "print(f\"By Severity: {analysis.get('by_severity', {}).get('counts')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Resolving Conflicts\n",
    "\n",
    "This is the critical step where we decide which value to trust. Semantica offers flexible resolution strategies.\n",
    "\n",
    "### Strategy A: Voting (Majority Rules)\n",
    "This strategy selects the value that appears most frequently. It is simple but treats all sources as equal.\n",
    "\n",
    "- Best when you have many independent sources of similar quality.\n",
    "- Less suitable if you have a single system-of-record that should always dominate.\n",
    "\n",
    "### Strategy B: Credibility Weighted\n",
    "This strategy calculates a weighted score for each value based on the `credibility` of its source. \n",
    "\n",
    "**Example:**\n",
    "*   `hr_db` (0.95) says \"1980-05-15\"\n",
    "*   `public_dir` (0.40) says \"1982-05-15\"\n",
    "\n",
    "Even if multiple low-quality sources agreed on the wrong date, the high-credibility source would likely win.\n",
    "\n",
    "**What the resolver returns:**\n",
    "\n",
    "- A list of resolution results where each item typically includes:\n",
    "  - whether it was resolved\n",
    "  - the chosen value (`resolved_value`)\n",
    "  - a confidence score\n",
    "  - metadata (like the property name) to support audit trails\n",
    "\n",
    "Run the next cell to compare voting vs credibility-weighted outcomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictResolver\n",
    "\n",
    "resolver = ConflictResolver()\n",
    "\n",
    "# CRITICAL: Link the source tracker to the resolver.\n",
    "# This allows the resolver to look up the credibility scores we registered in Step 2.\n",
    "resolver.set_source_tracker(source_tracker)\n",
    "\n",
    "print(\"--- Resolution: Voting ---\")\n",
    "voting_results = resolver.resolve_conflicts(conflicts, strategy=\"voting\")\n",
    "for res in voting_results:\n",
    "    print(f\"Property: {res.metadata.get('property_name'):<15} | Resolved Value: {res.resolved_value}\")\n",
    "\n",
    "print(\"\\n--- Resolution: Credibility Weighted ---\")\n",
    "# Notice how the HR DB's value is preferred due to higher credibility\n",
    "credibility_results = resolver.resolve_conflicts(conflicts, strategy=\"credibility_weighted\")\n",
    "for res in credibility_results:\n",
    "    print(f\"Property: {res.metadata.get('property_name'):<15} | Resolved Value: {res.resolved_value} (Confidence: {res.confidence:.2f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Generating Investigation Guides\n",
    "\n",
    "Not all conflicts can be resolved automatically. High-stakes or low-confidence resolutions require human review.\n",
    "\n",
    "The `InvestigationGuideGenerator` produces a structured \"flight plan\" for an analyst, detailing:\n",
    "\n",
    "- **What** is in conflict (entity + field + competing values).\n",
    "- **Who** is involved (which sources produced which values).\n",
    "- **How** to verify the correct data (actionable steps an analyst can follow).\n",
    "\n",
    "**When to generate guides:**\n",
    "\n",
    "- Low-confidence resolutions.\n",
    "- Conflicts on critical fields (identity, legal names, compliance attributes).\n",
    "- Any time you want a human-in-the-loop checkpoint before writing back to the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import InvestigationGuideGenerator\n",
    "\n",
    "guide_generator = InvestigationGuideGenerator()\n",
    "\n",
    "# Generate a guide for the first conflict (birth_date)\n",
    "guide = guide_generator.generate_guide(conflicts[0])\n",
    "\n",
    "print(f\"=== {guide.title} ===\")\n",
    "print(f\"Summary: {guide.conflict_summary}\\n\")\n",
    "\n",
    "print(\"Investigation Steps:\")\n",
    "for i, step in enumerate(guide.investigation_steps, 1):\n",
    "    print(f\"{i}. {step.description}\")\n",
    "    print(f\"   Action: {step.action}\")\n",
    "\n",
    "print(\"\\nRecommended Actions:\")\n",
    "for action in guide.recommended_actions:\n",
    "    print(f\"[ ] {action}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "You have successfully built a conflict resolution pipeline using Semantica! \n",
    "\n",
    "**Recap of what we achieved:**\n",
    "\n",
    "- **Simulated** multi-source entity records with realistic disagreements.\n",
    "- **Registered** sources with credibility scores to create a trust hierarchy.\n",
    "- **Detected** value conflicts for specific properties.\n",
    "- **Analyzed** conflicts to understand distribution by type and severity.\n",
    "- **Resolved** conflicts using voting and credibility-weighted strategies.\n",
    "- **Generated** an investigation guide to support human review.\n",
    "\n",
    "**Suggested next steps in a real project:**\n",
    "\n",
    "- Integrate with your ingestion layer so each extracted value includes a `source` and (ideally) a `timestamp`.\n",
    "- Expand detection beyond value conflicts using `ConflictDetector.detect_conflicts(...)`.\n",
    "- Store resolutions and guide outputs to build an audit trail for downstream consumers."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
