{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
    "\n",
    "#  Entity Extraction - Comprehensive Guide\n",
    "\n",
    "##  Overview\n",
    "\n",
    "This notebook provides a **comprehensive guide** to extracting named entities from text using Semantica's powerful NER (Named Entity Recognition) modules. You'll learn to use multiple extractors, methods, and advanced features to identify and classify entities in text.\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
    "\n",
    "###  Learning Objectives\n",
    "\n",
    "By the end of this notebook, you will be able to:\n",
    "\n",
    "-  Extract entities using `NERExtractor` and `NamedEntityRecognizer`\n",
    "-  Understand different extraction methods (pattern, regex, ML, HuggingFace, LLM)\n",
    "-  Use `EntityClassifier` to classify and group entities\n",
    "-  Apply `EntityConfidenceScorer` to assess extraction quality\n",
    "-  Create custom entity patterns with `CustomEntityDetector`\n",
    "-  Configure extraction parameters for optimal results\n",
    "-  Process multiple documents efficiently\n",
    "-  Handle edge cases and errors gracefully\n",
    "\n",
    "###  What You'll Learn\n",
    "\n",
    "| Component | Purpose | When to Use |\n",
    "|-----------|---------|-------------|\n",
    "| `NERExtractor` | Core entity extraction | Quick, simple extraction |\n",
    "| `NamedEntityRecognizer` | Advanced NER with configuration | Fine-tuned control needed |\n",
    "| `EntityClassifier` | Classify and group entities | Organizing extracted entities |\n",
    "| `EntityConfidenceScorer` | Score entity confidence | Quality assessment |\n",
    "| `CustomEntityDetector` | Domain-specific entities | Custom patterns needed |\n",
    "\n",
    "---\n",
    "\n",
    "##  Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install semantica\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 1: Basic Entity Extraction with NERExtractor\n",
    "\n",
    "Let's start with the simplest approach using `NERExtractor`. This class provides a straightforward interface for extracting named entities from text.\n",
    "\n",
    "### What is NERExtractor?\n",
    "\n",
    "`NERExtractor` is the core entity extraction class that:\n",
    "- Identifies named entities (people, organizations, locations, dates, etc.)\n",
    "- Returns entity objects with text, type, position, and confidence\n",
    "- Supports multiple extraction methods\n",
    "- Works out-of-the-box with sensible defaults"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "# Initialize the extractor\n",
    "ner_extractor = NERExtractor()\n",
    "\n",
    "# Sample text with various entity types\n",
    "text = \"\"\"\n",
    "Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
    "in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took \n",
    "over from Steve Jobs in August 2011. Apple is headquartered at One Apple Park Way in Cupertino.\n",
    "\"\"\"\n",
    "\n",
    "# Extract entities\n",
    "entities = ner_extractor.extract(text)\n",
    "\n",
    "print(f\" Extracted {len(entities)} entities:\\n\")\n",
    "print(\"-\" * 80)\n",
    "\n",
    "for i, entity in enumerate(entities, 1):\n",
    "    # Handle both dict and object formats\n",
    "    entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
    "    entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
    "    confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
    "    \n",
    "    print(f\"{i:2d}. {entity_text:30s} | Type: {entity_type:12s} | Confidence: {confidence:.2f}\")\n",
    "\n",
    "print(\"-\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Understanding Entity Objects\n",
    "\n",
    "Each extracted entity contains:\n",
    "\n",
    "| Attribute | Description | Example |\n",
    "|-----------|-------------|----------|\n",
    "| `text` | The entity text | \"Apple Inc.\" |\n",
    "| `label/type` | Entity category | \"ORG\" (Organization) |\n",
    "| `start_char` | Starting position | 0 |\n",
    "| `end_char` | Ending position | 10 |\n",
    "| `confidence` | Extraction confidence (0-1) | 0.95 |\n",
    "| `metadata` | Additional information | {\"method\": \"ml\"} |\n",
    "\n",
    "###  Common Entity Types\n",
    "\n",
    "- **PERSON**: People, including fictional characters\n",
    "- **ORG**: Companies, agencies, institutions\n",
    "- **GPE**: Countries, cities, states (Geo-Political Entities)\n",
    "- **LOC**: Non-GPE locations, mountain ranges, bodies of water\n",
    "- **DATE**: Absolute or relative dates or periods\n",
    "- **TIME**: Times smaller than a day\n",
    "- **MONEY**: Monetary values\n",
    "- **PERCENT**: Percentage values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 2: Visualizing Entities in Context\n",
    "\n",
    "Let's create a simple visualization to see entities highlighted in the original text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def highlight_entities(text, entities):\n",
    "    \"\"\"\n",
    "    Create a simple text visualization with entity markers.\n",
    "    \"\"\"\n",
    "    # Group entities by type\n",
    "    entity_types = {}\n",
    "    for entity in entities:\n",
    "        entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
    "        entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
    "        \n",
    "        if entity_type not in entity_types:\n",
    "            entity_types[entity_type] = []\n",
    "        entity_types[entity_type].append(entity_text)\n",
    "    \n",
    "    print(\"\\n Entity Visualization:\\n\")\n",
    "    print(\"=\" * 80)\n",
    "    \n",
    "    for entity_type, entity_list in sorted(entity_types.items()):\n",
    "        unique_entities = list(set(entity_list))\n",
    "        print(f\"\\n{entity_type}:\")\n",
    "        for ent in unique_entities:\n",
    "            print(f\"  • {ent}\")\n",
    "    \n",
    "    print(\"\\n\" + \"=\" * 80)\n",
    "\n",
    "# Visualize the extracted entities\n",
    "highlight_entities(text, entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ️ Step 3: Different Extraction Methods\n",
    "\n",
    "Semantica supports multiple extraction methods, each with different strengths:\n",
    "\n",
    "### Method Comparison\n",
    "\n",
    "| Method | Speed | Accuracy | Use Case | Requires |\n",
    "|--------|-------|----------|----------|----------|\n",
    "| **pattern** |  | ⭐⭐ | Simple, predictable patterns | Nothing |\n",
    "| **regex** |  | ⭐⭐⭐ | Custom patterns, IDs, codes | Regex knowledge |\n",
    "| **ml** (spaCy) |  | ⭐⭐⭐⭐ | General text, multiple languages | spaCy model |\n",
    "| **huggingface** |  | ⭐⭐⭐⭐⭐ | Domain-specific, fine-tuned | HF model |\n",
    "| **llm** |  | ⭐⭐⭐⭐⭐ | Complex, custom types | API key |\n",
    "\n",
    "Let's try different methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "sample_text = \"Apple Inc. was founded by Steve Jobs in Cupertino, California in 1976.\"\n",
    "\n",
    "print(\" Comparing Extraction Methods:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "# Try different methods\n",
    "methods_to_try = [\"pattern\", \"regex\", \"ml\"]\n",
    "\n",
    "for method_name in methods_to_try:\n",
    "    try:\n",
    "        print(f\"\\n Method: {method_name.upper()}\")\n",
    "        print(\"-\" * 40)\n",
    "        \n",
    "        extractor = NERExtractor(method=method_name)\n",
    "        entities = extractor.extract(sample_text)\n",
    "        \n",
    "        print(f\"Found {len(entities)} entities:\")\n",
    "        for entity in entities[:5]:  # Show first 5\n",
    "            entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
    "            entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
    "            print(f\"  • {entity_text} ({entity_type})\")\n",
    "            \n",
    "    except Exception as e:\n",
    "        print(f\"  ️  Method '{method_name}' not available: {str(e)[:50]}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 4: Advanced Entity Recognition with NamedEntityRecognizer\n",
    "\n",
    "`NamedEntityRecognizer` provides more control over the extraction process through configuration parameters.\n",
    "\n",
    "### Key Parameters:\n",
    "\n",
    "- **`methods`**: List of extraction methods to use (e.g., `[\"spacy\", \"rule-based\"]`)\n",
    "- **`confidence_threshold`**: Minimum confidence score (0.0-1.0, default: 0.5)\n",
    "- **`merge_overlapping`**: Whether to merge overlapping entities (default: True)\n",
    "- **`include_standard_types`**: Include standard entity types (PERSON, ORG, LOC, etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NamedEntityRecognizer\n",
    "\n",
    "# Create recognizer with custom configuration\n",
    "ner = NamedEntityRecognizer(\n",
    "    methods=[\"spacy\"],  # Use spaCy for ML-based extraction\n",
    "    confidence_threshold=0.7,  # Only keep high-confidence entities\n",
    "    merge_overlapping=True,  # Merge overlapping entity mentions\n",
    "    include_standard_types=True  # Include standard entity types\n",
    ")\n",
    "\n",
    "# Sample texts for batch processing\n",
    "texts = [\n",
    "    \"Tim Cook is the CEO of Apple Inc., based in Cupertino.\",\n",
    "    \"Microsoft Corporation, founded by Bill Gates, is headquartered in Redmond, Washington.\",\n",
    "    \"Amazon was founded by Jeff Bezos in Seattle in 1994.\",\n",
    "    \"Google was started by Larry Page and Sergey Brin at Stanford University.\"\n",
    "]\n",
    "\n",
    "print(\" Advanced Entity Recognition Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "all_entities = []\n",
    "for i, text in enumerate(texts, 1):\n",
    "    entities = ner.extract_entities(text)\n",
    "    all_entities.extend(entities)\n",
    "    \n",
    "    print(f\"\\n Text {i}: {text[:60]}...\")\n",
    "    print(f\"   Found {len(entities)} entities:\")\n",
    "    \n",
    "    for entity in entities:\n",
    "        entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
    "        entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
    "        confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
    "        print(f\"     • {entity_text:25s} | {entity_type:10s} | Confidence: {confidence:.2f}\")\n",
    "\n",
    "print(f\"\\n Total entities extracted: {len(all_entities)}\")\n",
    "print(\"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ️ Step 5: Entity Classification\n",
    "\n",
    "Use `EntityClassifier` to classify and group entities by type, and disambiguate similar entities.\n",
    "\n",
    "### What is EntityClassifier?\n",
    "\n",
    "The `EntityClassifier` helps you:\n",
    "- **Classify entities** by their type (normalize variations like \"ORG\" vs \"ORGANIZATION\")\n",
    "- **Group entities** by category for analysis\n",
    "- **Disambiguate entities** when multiple candidates exist\n",
    "- **Standardize entity types** across different extraction methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import EntityClassifier\n",
    "\n",
    "# Initialize classifier\n",
    "classifier = EntityClassifier()\n",
    "\n",
    "# Classify the entities we extracted earlier\n",
    "classified = classifier.classify_entities(all_entities)\n",
    "\n",
    "print(\"️  Entity Classification Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for entity_type, entity_list in sorted(classified.items()):\n",
    "    print(f\"\\n{entity_type} ({len(entity_list)} entities):\")\n",
    "    print(\"-\" * 40)\n",
    "    \n",
    "    # Get unique entity texts\n",
    "    unique_entities = set()\n",
    "    for entity in entity_list:\n",
    "        entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
    "        unique_entities.add(entity_text)\n",
    "    \n",
    "    for entity_text in sorted(unique_entities):\n",
    "        print(f\"  • {entity_text}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 6: Confidence Scoring\n",
    "\n",
    "Use `EntityConfidenceScorer` to assess and improve the confidence scores of extracted entities.\n",
    "\n",
    "### Why Confidence Scoring?\n",
    "\n",
    "Confidence scores help you:\n",
    "- **Filter low-quality extractions** (e.g., only keep entities with confidence > 0.8)\n",
    "- **Prioritize entities** for manual review or validation\n",
    "- **Assess extraction quality** across different methods or texts\n",
    "- **Make informed decisions** about which entities to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import EntityConfidenceScorer\n",
    "\n",
    "# Initialize confidence scorer\n",
    "scorer = EntityConfidenceScorer()\n",
    "\n",
    "# Score the entities\n",
    "scored_entities = scorer.score_entities(all_entities)\n",
    "\n",
    "print(\" Entity Confidence Scoring:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "# Group by confidence levels\n",
    "high_confidence = []\n",
    "medium_confidence = []\n",
    "low_confidence = []\n",
    "\n",
    "for entity in scored_entities:\n",
    "    confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
    "    \n",
    "    if confidence >= 0.8:\n",
    "        high_confidence.append(entity)\n",
    "    elif confidence >= 0.5:\n",
    "        medium_confidence.append(entity)\n",
    "    else:\n",
    "        low_confidence.append(entity)\n",
    "\n",
    "print(f\" High Confidence (≥0.8): {len(high_confidence)} entities\")\n",
    "print(f\"️  Medium Confidence (0.5-0.8): {len(medium_confidence)} entities\")\n",
    "print(f\" Low Confidence (<0.5): {len(low_confidence)} entities\")\n",
    "\n",
    "print(\"\\n Confidence Distribution:\")\n",
    "print(\"-\" * 40)\n",
    "\n",
    "# Show some examples from each category\n",
    "if high_confidence:\n",
    "    print(\"\\nHigh Confidence Examples:\")\n",
    "    for entity in high_confidence[:3]:\n",
    "        entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
    "        entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
    "        confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
    "        print(f\"  • {entity_text} ({entity_type}) - {confidence:.2f}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 7: Custom Entity Detection\n",
    "\n",
    "Use `CustomEntityDetector` to define domain-specific entity patterns.\n",
    "\n",
    "### When to Use Custom Patterns?\n",
    "\n",
    "Custom patterns are useful for:\n",
    "- **Domain-specific entities** (e.g., product codes, invoice numbers)\n",
    "- **Structured identifiers** (e.g., email addresses, phone numbers)\n",
    "- **Industry-specific terms** (e.g., medical codes, legal citations)\n",
    "- **Custom formats** not recognized by standard NER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import CustomEntityDetector\n",
    "import re\n",
    "\n",
    "# Define custom patterns\n",
    "custom_patterns = {\n",
    "    \"EMAIL\": r'\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b',\n",
    "    \"PHONE\": r'\\b\\d{3}[-.]?\\d{3}[-.]?\\d{4}\\b',\n",
    "    \"PRODUCT_CODE\": r'\\b[A-Z]{2,3}-\\d{4,6}\\b',\n",
    "    \"URL\": r'https?://[^\\s]+'\n",
    "}\n",
    "\n",
    "# Initialize custom detector\n",
    "custom_detector = CustomEntityDetector(patterns=custom_patterns)\n",
    "\n",
    "# Sample text with custom entities\n",
    "custom_text = \"\"\"\n",
    "For support, contact support@apple.com or call 1-800-692-7753.\n",
    "Order product SKU-12345 from https://store.apple.com.\n",
    "Technical inquiries: tech@apple.com or visit our website.\n",
    "\"\"\"\n",
    "\n",
    "print(\" Custom Entity Detection:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for entity_type in custom_patterns.keys():\n",
    "    entities = custom_detector.detect_custom_entities(custom_text, entity_type)\n",
    "    \n",
    "    if entities:\n",
    "        print(f\"\\n{entity_type}:\")\n",
    "        for entity in entities:\n",
    "            entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
    "            print(f\"  • {entity_text}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 8: Batch Processing\n",
    "\n",
    "Process multiple documents efficiently using batch processing capabilities.\n",
    "\n",
    "### Benefits of Batch Processing:\n",
    "\n",
    "- **Performance**: Process multiple documents in one call\n",
    "- **Consistency**: Same configuration applied to all documents\n",
    "- **Efficiency**: Reduced overhead from initialization\n",
    "- **Scalability**: Handle large document collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample document collection\n",
    "documents = [\n",
    "    \"Apple Inc. released the iPhone 15 in September 2023.\",\n",
    "    \"Microsoft announced Azure AI updates at Build 2023 in Seattle.\",\n",
    "    \"Google's Sundar Pichai spoke at I/O 2023 in Mountain View, California.\",\n",
    "    \"Tesla's Elon Musk unveiled the Cybertruck in Austin, Texas.\",\n",
    "    \"Amazon Web Services launched new features in Northern Virginia.\"\n",
    "]\n",
    "\n",
    "print(\" Batch Processing Results:\\n\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "# Process all documents\n",
    "batch_results = ner.process_batch(documents)\n",
    "\n",
    "# Analyze results\n",
    "total_entities = 0\n",
    "entity_type_counts = {}\n",
    "\n",
    "for i, (doc, entities) in enumerate(zip(documents, batch_results), 1):\n",
    "    total_entities += len(entities)\n",
    "    \n",
    "    print(f\"\\n Document {i}:\")\n",
    "    print(f\"   Text: {doc[:50]}...\")\n",
    "    print(f\"   Entities: {len(entities)}\")\n",
    "    \n",
    "    for entity in entities:\n",
    "        entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
    "        entity_type_counts[entity_type] = entity_type_counts.get(entity_type, 0) + 1\n",
    "\n",
    "print(f\"\\n Batch Processing Summary:\")\n",
    "print(\"-\" * 40)\n",
    "print(f\"Documents processed: {len(documents)}\")\n",
    "print(f\"Total entities: {total_entities}\")\n",
    "print(f\"Average per document: {total_entities/len(documents):.1f}\")\n",
    "\n",
    "print(\"\\n Entity Type Distribution:\")\n",
    "for entity_type, count in sorted(entity_type_counts.items(), key=lambda x: x[1], reverse=True):\n",
    "    print(f\"  {entity_type}: {count}\")\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 9: Best Practices & Tips\n",
    "\n",
    "###  Choosing the Right Method\n",
    "\n",
    "1. **Start with ML (spaCy)** for general text\n",
    "2. **Use patterns/regex** for structured data (IDs, codes)\n",
    "3. **Try HuggingFace** for domain-specific needs\n",
    "4. **Consider LLM** for complex, custom entity types\n",
    "\n",
    "### ️ Optimizing Performance\n",
    "\n",
    "- **Set appropriate confidence thresholds** (0.7-0.8 for production)\n",
    "- **Use batch processing** for multiple documents\n",
    "- **Enable merge_overlapping** to reduce duplicates\n",
    "- **Cache extractors** instead of recreating them\n",
    "\n",
    "###  Common Pitfalls to Avoid\n",
    "\n",
    "-  **Don't** use very low confidence thresholds (< 0.5)\n",
    "-  **Don't** process one document at a time in loops\n",
    "-  **Don't** ignore entity metadata (contains useful info)\n",
    "-  **Don't** forget to handle extraction errors\n",
    "\n",
    "###  When to Use Each Class\n",
    "\n",
    "| Use Case | Recommended Class |\n",
    "|----------|-------------------|\n",
    "| Quick extraction | `NERExtractor` |\n",
    "| Fine-tuned control | `NamedEntityRecognizer` |\n",
    "| Grouping entities | `EntityClassifier` |\n",
    "| Quality assessment | `EntityConfidenceScorer` |\n",
    "| Domain-specific | `CustomEntityDetector` |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Summary\n",
    "\n",
    "### What You've Learned\n",
    "\n",
    "In this notebook, you've learned how to:\n",
    "\n",
    " **Extract entities** using `NERExtractor` and `NamedEntityRecognizer`  \n",
    " **Compare different extraction methods** (pattern, regex, ML, HuggingFace, LLM)  \n",
    " **Classify and group entities** with `EntityClassifier`  \n",
    " **Score entity confidence** using `EntityConfidenceScorer`  \n",
    " **Create custom patterns** with `CustomEntityDetector`  \n",
    " **Process documents in batch** for efficiency  \n",
    " **Apply best practices** for production use  \n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "1. **Multiple methods available**: Choose based on your needs (speed vs accuracy)\n",
    "2. **Configuration matters**: Tune parameters for optimal results\n",
    "3. **Confidence is key**: Use thresholds to filter low-quality extractions\n",
    "4. **Custom patterns work**: For domain-specific entities\n",
    "5. **Batch processing scales**: Process multiple documents efficiently\n",
    "\n",
    "### Next Steps\n",
    "\n",
    " **Next Notebook**: [06_Relation_Extraction.ipynb](./06_Relation_Extraction.ipynb)  \n",
    "Learn how to extract relationships between the entities you've identified!\n",
    "\n",
    " **Further Reading**:\n",
    "- [Semantic Extract API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
    "- [Advanced Extraction Techniques](../advanced/01_Advanced_Extraction.ipynb)\n",
    "- [Building Knowledge Graphs](./07_Building_Knowledge_Graphs.ipynb)\n",
    "\n",
    "---\n",
    "\n",
    "**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
