mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
KaifAhmad1
7a6f1d0417
Add a Cite Us section to the README with BibTeX citation info, and align it with docs/citation.md (author/organization: Semantica, 2026). Update LICENSE and docs/project-license.md copyright holder to Semantica, and replace the stale Hawksight-AI GitHub org slug with semantica-agi across READMEs, plugin manifests, cookbook notebooks, and GitHub templates.
651 lines
25 KiB
Plaintext
651 lines
25 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"[](https://colab.research.google.com/github/semantica-agi/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
|
||
"\n",
|
||
"# Entity Extraction - Comprehensive Guide\n",
|
||
"\n",
|
||
"## Overview\n",
|
||
"\n",
|
||
"This notebook provides a **comprehensive guide** to extracting named entities from text using Semantica's powerful NER (Named Entity Recognition) modules. You'll learn to use multiple extractors, methods, and advanced features to identify and classify entities in text.\n",
|
||
"\n",
|
||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
|
||
"\n",
|
||
"### Learning Objectives\n",
|
||
"\n",
|
||
"By the end of this notebook, you will be able to:\n",
|
||
"\n",
|
||
"- Extract entities using `NERExtractor` and `NamedEntityRecognizer`\n",
|
||
"- Understand different extraction methods (pattern, regex, ML, HuggingFace, LLM)\n",
|
||
"- Use `EntityClassifier` to classify and group entities\n",
|
||
"- Apply `EntityConfidenceScorer` to assess extraction quality\n",
|
||
"- Create custom entity patterns with `CustomEntityDetector`\n",
|
||
"- Configure extraction parameters for optimal results\n",
|
||
"- Process multiple documents efficiently\n",
|
||
"- Handle edge cases and errors gracefully\n",
|
||
"\n",
|
||
"### What You'll Learn\n",
|
||
"\n",
|
||
"| Component | Purpose | When to Use |\n",
|
||
"|-----------|---------|-------------|\n",
|
||
"| `NERExtractor` | Core entity extraction | Quick, simple extraction |\n",
|
||
"| `NamedEntityRecognizer` | Advanced NER with configuration | Fine-tuned control needed |\n",
|
||
"| `EntityClassifier` | Classify and group entities | Organizing extracted entities |\n",
|
||
"| `EntityConfidenceScorer` | Score entity confidence | Quality assessment |\n",
|
||
"| `CustomEntityDetector` | Domain-specific entities | Custom patterns needed |\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"## Installation\n",
|
||
"\n",
|
||
"Install Semantica from PyPI:\n",
|
||
"\n",
|
||
"```bash\n",
|
||
"pip install semantica\n",
|
||
"# Or with all optional dependencies:\n",
|
||
"pip install semantica[all]\n",
|
||
"```\n",
|
||
"\n",
|
||
"---"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install semantica\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 1: Basic Entity Extraction with NERExtractor\n",
|
||
"\n",
|
||
"Let's start with the simplest approach using `NERExtractor`. This class provides a straightforward interface for extracting named entities from text.\n",
|
||
"\n",
|
||
"### What is NERExtractor?\n",
|
||
"\n",
|
||
"`NERExtractor` is the core entity extraction class that:\n",
|
||
"- Identifies named entities (people, organizations, locations, dates, etc.)\n",
|
||
"- Returns entity objects with text, type, position, and confidence\n",
|
||
"- Supports multiple extraction methods\n",
|
||
"- Works out-of-the-box with sensible defaults"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from semantica.semantic_extract import NERExtractor\n",
|
||
"\n",
|
||
"# Initialize the extractor\n",
|
||
"ner_extractor = NERExtractor()\n",
|
||
"\n",
|
||
"# Sample text with various entity types\n",
|
||
"text = \"\"\"\n",
|
||
"Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
|
||
"in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took \n",
|
||
"over from Steve Jobs in August 2011. Apple is headquartered at One Apple Park Way in Cupertino.\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"# Extract entities\n",
|
||
"entities = ner_extractor.extract(text)\n",
|
||
"\n",
|
||
"print(f\" Extracted {len(entities)} entities:\\n\")\n",
|
||
"print(\"-\" * 80)\n",
|
||
"\n",
|
||
"for i, entity in enumerate(entities, 1):\n",
|
||
" # Handle both dict and object formats\n",
|
||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||
" \n",
|
||
" print(f\"{i:2d}. {entity_text:30s} | Type: {entity_type:12s} | Confidence: {confidence:.2f}\")\n",
|
||
"\n",
|
||
"print(\"-\" * 80)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Understanding Entity Objects\n",
|
||
"\n",
|
||
"Each extracted entity contains:\n",
|
||
"\n",
|
||
"| Attribute | Description | Example |\n",
|
||
"|-----------|-------------|----------|\n",
|
||
"| `text` | The entity text | \"Apple Inc.\" |\n",
|
||
"| `label/type` | Entity category | \"ORG\" (Organization) |\n",
|
||
"| `start_char` | Starting position | 0 |\n",
|
||
"| `end_char` | Ending position | 10 |\n",
|
||
"| `confidence` | Extraction confidence (0-1) | 0.95 |\n",
|
||
"| `metadata` | Additional information | {\"method\": \"ml\"} |\n",
|
||
"\n",
|
||
"### Common Entity Types\n",
|
||
"\n",
|
||
"- **PERSON**: People, including fictional characters\n",
|
||
"- **ORG**: Companies, agencies, institutions\n",
|
||
"- **GPE**: Countries, cities, states (Geo-Political Entities)\n",
|
||
"- **LOC**: Non-GPE locations, mountain ranges, bodies of water\n",
|
||
"- **DATE**: Absolute or relative dates or periods\n",
|
||
"- **TIME**: Times smaller than a day\n",
|
||
"- **MONEY**: Monetary values\n",
|
||
"- **PERCENT**: Percentage values"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 2: Visualizing Entities in Context\n",
|
||
"\n",
|
||
"Let's create a simple visualization to see entities highlighted in the original text."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def highlight_entities(text, entities):\n",
|
||
" \"\"\"\n",
|
||
" Create a simple text visualization with entity markers.\n",
|
||
" \"\"\"\n",
|
||
" # Group entities by type\n",
|
||
" entity_types = {}\n",
|
||
" for entity in entities:\n",
|
||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||
" \n",
|
||
" if entity_type not in entity_types:\n",
|
||
" entity_types[entity_type] = []\n",
|
||
" entity_types[entity_type].append(entity_text)\n",
|
||
" \n",
|
||
" print(\"\\n Entity Visualization:\\n\")\n",
|
||
" print(\"=\" * 80)\n",
|
||
" \n",
|
||
" for entity_type, entity_list in sorted(entity_types.items()):\n",
|
||
" unique_entities = list(set(entity_list))\n",
|
||
" print(f\"\\n{entity_type}:\")\n",
|
||
" for ent in unique_entities:\n",
|
||
" print(f\" • {ent}\")\n",
|
||
" \n",
|
||
" print(\"\\n\" + \"=\" * 80)\n",
|
||
"\n",
|
||
"# Visualize the extracted entities\n",
|
||
"highlight_entities(text, entities)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## ️ Step 3: Different Extraction Methods\n",
|
||
"\n",
|
||
"Semantica supports multiple extraction methods, each with different strengths:\n",
|
||
"\n",
|
||
"### Method Comparison\n",
|
||
"\n",
|
||
"| Method | Speed | Accuracy | Use Case | Requires |\n",
|
||
"|--------|-------|----------|----------|----------|\n",
|
||
"| **pattern** | | ⭐⭐ | Simple, predictable patterns | Nothing |\n",
|
||
"| **regex** | | ⭐⭐⭐ | Custom patterns, IDs, codes | Regex knowledge |\n",
|
||
"| **ml** (spaCy) | | ⭐⭐⭐⭐ | General text, multiple languages | spaCy model |\n",
|
||
"| **huggingface** | | ⭐⭐⭐⭐⭐ | Domain-specific, fine-tuned | HF model |\n",
|
||
"| **llm** | | ⭐⭐⭐⭐⭐ | Complex, custom types | API key |\n",
|
||
"\n",
|
||
"Let's try different methods:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from semantica.semantic_extract import NERExtractor\n",
|
||
"\n",
|
||
"sample_text = \"Apple Inc. was founded by Steve Jobs in Cupertino, California in 1976.\"\n",
|
||
"\n",
|
||
"print(\" Comparing Extraction Methods:\\n\")\n",
|
||
"print(\"=\" * 80)\n",
|
||
"\n",
|
||
"# Try different methods\n",
|
||
"methods_to_try = [\"pattern\", \"regex\", \"ml\"]\n",
|
||
"\n",
|
||
"for method_name in methods_to_try:\n",
|
||
" try:\n",
|
||
" print(f\"\\n Method: {method_name.upper()}\")\n",
|
||
" print(\"-\" * 40)\n",
|
||
" \n",
|
||
" extractor = NERExtractor(method=method_name)\n",
|
||
" entities = extractor.extract(sample_text)\n",
|
||
" \n",
|
||
" print(f\"Found {len(entities)} entities:\")\n",
|
||
" for entity in entities[:5]: # Show first 5\n",
|
||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||
" print(f\" • {entity_text} ({entity_type})\")\n",
|
||
" \n",
|
||
" except Exception as e:\n",
|
||
" print(f\" ️ Method '{method_name}' not available: {str(e)[:50]}\")\n",
|
||
"\n",
|
||
"print(\"\\n\" + \"=\" * 80)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 4: Advanced Entity Recognition with NamedEntityRecognizer\n",
|
||
"\n",
|
||
"`NamedEntityRecognizer` provides more control over the extraction process through configuration parameters.\n",
|
||
"\n",
|
||
"### Key Parameters:\n",
|
||
"\n",
|
||
"- **`methods`**: List of extraction methods to use (e.g., `[\"spacy\", \"rule-based\"]`)\n",
|
||
"- **`confidence_threshold`**: Minimum confidence score (0.0-1.0, default: 0.5)\n",
|
||
"- **`merge_overlapping`**: Whether to merge overlapping entities (default: True)\n",
|
||
"- **`include_standard_types`**: Include standard entity types (PERSON, ORG, LOC, etc.)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from semantica.semantic_extract import NamedEntityRecognizer\n",
|
||
"\n",
|
||
"# Create recognizer with custom configuration\n",
|
||
"ner = NamedEntityRecognizer(\n",
|
||
" methods=[\"spacy\"], # Use spaCy for ML-based extraction\n",
|
||
" confidence_threshold=0.7, # Only keep high-confidence entities\n",
|
||
" merge_overlapping=True, # Merge overlapping entity mentions\n",
|
||
" include_standard_types=True # Include standard entity types\n",
|
||
")\n",
|
||
"\n",
|
||
"# Sample texts for batch processing\n",
|
||
"texts = [\n",
|
||
" \"Tim Cook is the CEO of Apple Inc., based in Cupertino.\",\n",
|
||
" \"Microsoft Corporation, founded by Bill Gates, is headquartered in Redmond, Washington.\",\n",
|
||
" \"Amazon was founded by Jeff Bezos in Seattle in 1994.\",\n",
|
||
" \"Google was started by Larry Page and Sergey Brin at Stanford University.\"\n",
|
||
"]\n",
|
||
"\n",
|
||
"print(\" Advanced Entity Recognition Results:\\n\")\n",
|
||
"print(\"=\" * 80)\n",
|
||
"\n",
|
||
"all_entities = []\n",
|
||
"for i, text in enumerate(texts, 1):\n",
|
||
" entities = ner.extract_entities(text)\n",
|
||
" all_entities.extend(entities)\n",
|
||
" \n",
|
||
" print(f\"\\n Text {i}: {text[:60]}...\")\n",
|
||
" print(f\" Found {len(entities)} entities:\")\n",
|
||
" \n",
|
||
" for entity in entities:\n",
|
||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||
" print(f\" • {entity_text:25s} | {entity_type:10s} | Confidence: {confidence:.2f}\")\n",
|
||
"\n",
|
||
"print(f\"\\n Total entities extracted: {len(all_entities)}\")\n",
|
||
"print(\"=\" * 80)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## ️ Step 5: Entity Classification\n",
|
||
"\n",
|
||
"Use `EntityClassifier` to classify and group entities by type, and disambiguate similar entities.\n",
|
||
"\n",
|
||
"### What is EntityClassifier?\n",
|
||
"\n",
|
||
"The `EntityClassifier` helps you:\n",
|
||
"- **Classify entities** by their type (normalize variations like \"ORG\" vs \"ORGANIZATION\")\n",
|
||
"- **Group entities** by category for analysis\n",
|
||
"- **Disambiguate entities** when multiple candidates exist\n",
|
||
"- **Standardize entity types** across different extraction methods"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from semantica.semantic_extract import EntityClassifier\n",
|
||
"\n",
|
||
"# Initialize classifier\n",
|
||
"classifier = EntityClassifier()\n",
|
||
"\n",
|
||
"# Classify the entities we extracted earlier\n",
|
||
"classified = classifier.classify_entities(all_entities)\n",
|
||
"\n",
|
||
"print(\"️ Entity Classification Results:\\n\")\n",
|
||
"print(\"=\" * 80)\n",
|
||
"\n",
|
||
"for entity_type, entity_list in sorted(classified.items()):\n",
|
||
" print(f\"\\n{entity_type} ({len(entity_list)} entities):\")\n",
|
||
" print(\"-\" * 40)\n",
|
||
" \n",
|
||
" # Get unique entity texts\n",
|
||
" unique_entities = set()\n",
|
||
" for entity in entity_list:\n",
|
||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||
" unique_entities.add(entity_text)\n",
|
||
" \n",
|
||
" for entity_text in sorted(unique_entities):\n",
|
||
" print(f\" • {entity_text}\")\n",
|
||
"\n",
|
||
"print(\"\\n\" + \"=\" * 80)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 6: Confidence Scoring\n",
|
||
"\n",
|
||
"Use `EntityConfidenceScorer` to assess and improve the confidence scores of extracted entities.\n",
|
||
"\n",
|
||
"### Why Confidence Scoring?\n",
|
||
"\n",
|
||
"Confidence scores help you:\n",
|
||
"- **Filter low-quality extractions** (e.g., only keep entities with confidence > 0.8)\n",
|
||
"- **Prioritize entities** for manual review or validation\n",
|
||
"- **Assess extraction quality** across different methods or texts\n",
|
||
"- **Make informed decisions** about which entities to use"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from semantica.semantic_extract import EntityConfidenceScorer\n",
|
||
"\n",
|
||
"# Initialize confidence scorer\n",
|
||
"scorer = EntityConfidenceScorer()\n",
|
||
"\n",
|
||
"# Score the entities\n",
|
||
"scored_entities = scorer.score_entities(all_entities)\n",
|
||
"\n",
|
||
"print(\" Entity Confidence Scoring:\\n\")\n",
|
||
"print(\"=\" * 80)\n",
|
||
"\n",
|
||
"# Group by confidence levels\n",
|
||
"high_confidence = []\n",
|
||
"medium_confidence = []\n",
|
||
"low_confidence = []\n",
|
||
"\n",
|
||
"for entity in scored_entities:\n",
|
||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||
" \n",
|
||
" if confidence >= 0.8:\n",
|
||
" high_confidence.append(entity)\n",
|
||
" elif confidence >= 0.5:\n",
|
||
" medium_confidence.append(entity)\n",
|
||
" else:\n",
|
||
" low_confidence.append(entity)\n",
|
||
"\n",
|
||
"print(f\" High Confidence (≥0.8): {len(high_confidence)} entities\")\n",
|
||
"print(f\"️ Medium Confidence (0.5-0.8): {len(medium_confidence)} entities\")\n",
|
||
"print(f\" Low Confidence (<0.5): {len(low_confidence)} entities\")\n",
|
||
"\n",
|
||
"print(\"\\n Confidence Distribution:\")\n",
|
||
"print(\"-\" * 40)\n",
|
||
"\n",
|
||
"# Show some examples from each category\n",
|
||
"if high_confidence:\n",
|
||
" print(\"\\nHigh Confidence Examples:\")\n",
|
||
" for entity in high_confidence[:3]:\n",
|
||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||
" print(f\" • {entity_text} ({entity_type}) - {confidence:.2f}\")\n",
|
||
"\n",
|
||
"print(\"\\n\" + \"=\" * 80)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 7: Custom Entity Detection\n",
|
||
"\n",
|
||
"Use `CustomEntityDetector` to define domain-specific entity patterns.\n",
|
||
"\n",
|
||
"### When to Use Custom Patterns?\n",
|
||
"\n",
|
||
"Custom patterns are useful for:\n",
|
||
"- **Domain-specific entities** (e.g., product codes, invoice numbers)\n",
|
||
"- **Structured identifiers** (e.g., email addresses, phone numbers)\n",
|
||
"- **Industry-specific terms** (e.g., medical codes, legal citations)\n",
|
||
"- **Custom formats** not recognized by standard NER"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from semantica.semantic_extract import CustomEntityDetector\n",
|
||
"import re\n",
|
||
"\n",
|
||
"# Define custom patterns\n",
|
||
"custom_patterns = {\n",
|
||
" \"EMAIL\": r'\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b',\n",
|
||
" \"PHONE\": r'\\b\\d{3}[-.]?\\d{3}[-.]?\\d{4}\\b',\n",
|
||
" \"PRODUCT_CODE\": r'\\b[A-Z]{2,3}-\\d{4,6}\\b',\n",
|
||
" \"URL\": r'https?://[^\\s]+'\n",
|
||
"}\n",
|
||
"\n",
|
||
"# Initialize custom detector\n",
|
||
"custom_detector = CustomEntityDetector(patterns=custom_patterns)\n",
|
||
"\n",
|
||
"# Sample text with custom entities\n",
|
||
"custom_text = \"\"\"\n",
|
||
"For support, contact support@apple.com or call 1-800-692-7753.\n",
|
||
"Order product SKU-12345 from https://store.apple.com.\n",
|
||
"Technical inquiries: tech@apple.com or visit our website.\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"print(\" Custom Entity Detection:\\n\")\n",
|
||
"print(\"=\" * 80)\n",
|
||
"\n",
|
||
"for entity_type in custom_patterns.keys():\n",
|
||
" entities = custom_detector.detect_custom_entities(custom_text, entity_type)\n",
|
||
" \n",
|
||
" if entities:\n",
|
||
" print(f\"\\n{entity_type}:\")\n",
|
||
" for entity in entities:\n",
|
||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||
" print(f\" • {entity_text}\")\n",
|
||
"\n",
|
||
"print(\"\\n\" + \"=\" * 80)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 8: Batch Processing\n",
|
||
"\n",
|
||
"Process multiple documents efficiently using batch processing capabilities.\n",
|
||
"\n",
|
||
"### Benefits of Batch Processing:\n",
|
||
"\n",
|
||
"- **Performance**: Process multiple documents in one call\n",
|
||
"- **Consistency**: Same configuration applied to all documents\n",
|
||
"- **Efficiency**: Reduced overhead from initialization\n",
|
||
"- **Scalability**: Handle large document collections"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Sample document collection\n",
|
||
"documents = [\n",
|
||
" \"Apple Inc. released the iPhone 15 in September 2023.\",\n",
|
||
" \"Microsoft announced Azure AI updates at Build 2023 in Seattle.\",\n",
|
||
" \"Google's Sundar Pichai spoke at I/O 2023 in Mountain View, California.\",\n",
|
||
" \"Tesla's Elon Musk unveiled the Cybertruck in Austin, Texas.\",\n",
|
||
" \"Amazon Web Services launched new features in Northern Virginia.\"\n",
|
||
"]\n",
|
||
"\n",
|
||
"print(\" Batch Processing Results:\\n\")\n",
|
||
"print(\"=\" * 80)\n",
|
||
"\n",
|
||
"# Process all documents\n",
|
||
"batch_results = ner.process_batch(documents)\n",
|
||
"\n",
|
||
"# Analyze results\n",
|
||
"total_entities = 0\n",
|
||
"entity_type_counts = {}\n",
|
||
"\n",
|
||
"for i, (doc, entities) in enumerate(zip(documents, batch_results), 1):\n",
|
||
" total_entities += len(entities)\n",
|
||
" \n",
|
||
" print(f\"\\n Document {i}:\")\n",
|
||
" print(f\" Text: {doc[:50]}...\")\n",
|
||
" print(f\" Entities: {len(entities)}\")\n",
|
||
" \n",
|
||
" for entity in entities:\n",
|
||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||
" entity_type_counts[entity_type] = entity_type_counts.get(entity_type, 0) + 1\n",
|
||
"\n",
|
||
"print(f\"\\n Batch Processing Summary:\")\n",
|
||
"print(\"-\" * 40)\n",
|
||
"print(f\"Documents processed: {len(documents)}\")\n",
|
||
"print(f\"Total entities: {total_entities}\")\n",
|
||
"print(f\"Average per document: {total_entities/len(documents):.1f}\")\n",
|
||
"\n",
|
||
"print(\"\\n Entity Type Distribution:\")\n",
|
||
"for entity_type, count in sorted(entity_type_counts.items(), key=lambda x: x[1], reverse=True):\n",
|
||
" print(f\" {entity_type}: {count}\")\n",
|
||
"\n",
|
||
"print(\"\\n\" + \"=\" * 80)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Step 9: Best Practices & Tips\n",
|
||
"\n",
|
||
"### Choosing the Right Method\n",
|
||
"\n",
|
||
"1. **Start with ML (spaCy)** for general text\n",
|
||
"2. **Use patterns/regex** for structured data (IDs, codes)\n",
|
||
"3. **Try HuggingFace** for domain-specific needs\n",
|
||
"4. **Consider LLM** for complex, custom entity types\n",
|
||
"\n",
|
||
"### ️ Optimizing Performance\n",
|
||
"\n",
|
||
"- **Set appropriate confidence thresholds** (0.7-0.8 for production)\n",
|
||
"- **Use batch processing** for multiple documents\n",
|
||
"- **Enable merge_overlapping** to reduce duplicates\n",
|
||
"- **Cache extractors** instead of recreating them\n",
|
||
"\n",
|
||
"### Common Pitfalls to Avoid\n",
|
||
"\n",
|
||
"- **Don't** use very low confidence thresholds (< 0.5)\n",
|
||
"- **Don't** process one document at a time in loops\n",
|
||
"- **Don't** ignore entity metadata (contains useful info)\n",
|
||
"- **Don't** forget to handle extraction errors\n",
|
||
"\n",
|
||
"### When to Use Each Class\n",
|
||
"\n",
|
||
"| Use Case | Recommended Class |\n",
|
||
"|----------|-------------------|\n",
|
||
"| Quick extraction | `NERExtractor` |\n",
|
||
"| Fine-tuned control | `NamedEntityRecognizer` |\n",
|
||
"| Grouping entities | `EntityClassifier` |\n",
|
||
"| Quality assessment | `EntityConfidenceScorer` |\n",
|
||
"| Domain-specific | `CustomEntityDetector` |"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Summary\n",
|
||
"\n",
|
||
"### What You've Learned\n",
|
||
"\n",
|
||
"In this notebook, you've learned how to:\n",
|
||
"\n",
|
||
" **Extract entities** using `NERExtractor` and `NamedEntityRecognizer` \n",
|
||
" **Compare different extraction methods** (pattern, regex, ML, HuggingFace, LLM) \n",
|
||
" **Classify and group entities** with `EntityClassifier` \n",
|
||
" **Score entity confidence** using `EntityConfidenceScorer` \n",
|
||
" **Create custom patterns** with `CustomEntityDetector` \n",
|
||
" **Process documents in batch** for efficiency \n",
|
||
" **Apply best practices** for production use \n",
|
||
"\n",
|
||
"### Key Takeaways\n",
|
||
"\n",
|
||
"1. **Multiple methods available**: Choose based on your needs (speed vs accuracy)\n",
|
||
"2. **Configuration matters**: Tune parameters for optimal results\n",
|
||
"3. **Confidence is key**: Use thresholds to filter low-quality extractions\n",
|
||
"4. **Custom patterns work**: For domain-specific entities\n",
|
||
"5. **Batch processing scales**: Process multiple documents efficiently\n",
|
||
"\n",
|
||
"### Next Steps\n",
|
||
"\n",
|
||
" **Next Notebook**: [06_Relation_Extraction.ipynb](./06_Relation_Extraction.ipynb) \n",
|
||
"Learn how to extract relationships between the entities you've identified!\n",
|
||
"\n",
|
||
" **Further Reading**:\n",
|
||
"- [Semantic Extract API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
|
||
"- [Advanced Extraction Techniques](../advanced/01_Advanced_Extraction.ipynb)\n",
|
||
"- [Building Knowledge Graphs](./07_Building_Knowledge_Graphs.ipynb)\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/semantica-agi/semantica) or [documentation](https://semantica.readthedocs.io)."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.9"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|