mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-10 04:00:35 +00:00
refactor(semantic_extract): remove build function and enhance documentation
BREAKING CHANGE: Removed build() convenience function from semantic_extract module - Removed build() function from semantic_extract/__init__.py - Updated __all__ exports to remove 'build' - Resolved merge conflicts in named_entity_recognizer.py, relation_extractor.py, triple_extractor.py - Updated semantic_extract_usage.md with class-based examples - Updated docs/reference/semantic_extract.md with detailed parameter documentation - Fixed 01_GraphRAG_Complete.ipynb to use individual extractor classes - Enhanced 05_Entity_Extraction.ipynb with comprehensive examples (9 sections) - Enhanced 06_Relation_Extraction.ipynb with complete pipeline examples (9 sections) Users should now use individual classes (NERExtractor, RelationExtractor, TripleExtractor, etc.) instead of the build() function for better control and flexibility. Migration guide available in documentation.
This commit is contained in:
KaifAhmad1
@@ -4,24 +4,42 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Entity_Extraction.ipynb)\n",
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)\n",
|
||||
"\n",
|
||||
"# Entity Extraction\n",
|
||||
"# Entity Extraction - Comprehensive Guide\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to extract named entities from text using Semantica's NER modules. You'll learn to use `NERExtractor` and `NamedEntityRecognizer` to identify entities in text.\n",
|
||||
"This notebook provides a **comprehensive guide** to extracting named entities from text using Semantica's powerful NER (Named Entity Recognition) modules. You'll learn to use multiple extractors, methods, and advanced features to identify and classify entities in text.\n",
|
||||
"\n",
|
||||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
|
||||
"\n",
|
||||
"### Learning Objectives\n",
|
||||
"### Learning Objectives\n",
|
||||
"\n",
|
||||
"- Use `NERExtractor` to extract entities from text\n",
|
||||
"- Use `NamedEntityRecognizer` for advanced entity recognition\n",
|
||||
"- Understand entity types and confidence scores\n",
|
||||
"- Extract entities from multiple documents\n",
|
||||
"By the end of this notebook, you will be able to:\n",
|
||||
"\n",
|
||||
"## Installation\n",
|
||||
"- Extract entities using `NERExtractor` and `NamedEntityRecognizer`\n",
|
||||
"- Understand different extraction methods (pattern, regex, ML, HuggingFace, LLM)\n",
|
||||
"- Use `EntityClassifier` to classify and group entities\n",
|
||||
"- Apply `EntityConfidenceScorer` to assess extraction quality\n",
|
||||
"- Create custom entity patterns with `CustomEntityDetector`\n",
|
||||
"- Configure extraction parameters for optimal results\n",
|
||||
"- Process multiple documents efficiently\n",
|
||||
"- Handle edge cases and errors gracefully\n",
|
||||
"\n",
|
||||
"### What You'll Learn\n",
|
||||
"\n",
|
||||
"| Component | Purpose | When to Use |\n",
|
||||
"|-----------|---------|-------------|\n",
|
||||
"| `NERExtractor` | Core entity extraction | Quick, simple extraction |\n",
|
||||
"| `NamedEntityRecognizer` | Advanced NER with configuration | Fine-tuned control needed |\n",
|
||||
"| `EntityClassifier` | Classify and group entities | Organizing extracted entities |\n",
|
||||
"| `EntityConfidenceScorer` | Score entity confidence | Quality assessment |\n",
|
||||
"| `CustomEntityDetector` | Domain-specific entities | Custom patterns needed |\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install Semantica from PyPI:\n",
|
||||
"\n",
|
||||
@@ -31,11 +49,24 @@
|
||||
"pip install semantica[all]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 1: Basic Entity Extraction with NERExtractor\n",
|
||||
"\n",
|
||||
"## Step 1: Basic Entity Extraction\n",
|
||||
"Let's start with the simplest approach using `NERExtractor`. This class provides a straightforward interface for extracting named entities from text.\n",
|
||||
"\n",
|
||||
"Extract entities using NERExtractor.\n"
|
||||
"### What is NERExtractor?\n",
|
||||
"\n",
|
||||
"`NERExtractor` is the core entity extraction class that:\n",
|
||||
"- Identifies named entities (people, organizations, locations, dates, etc.)\n",
|
||||
"- Returns entity objects with text, type, position, and confidence\n",
|
||||
"- Supports multiple extraction methods\n",
|
||||
"- Works out-of-the-box with sensible defaults"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -46,26 +77,177 @@
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NERExtractor\n",
|
||||
"\n",
|
||||
"# Initialize the extractor\n",
|
||||
"ner_extractor = NERExtractor()\n",
|
||||
"\n",
|
||||
"text = \"Apple Inc. is a technology company founded by Steve Jobs in Cupertino, California in 1976.\"\n",
|
||||
"# Sample text with various entity types\n",
|
||||
"text = \"\"\"\n",
|
||||
"Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne \n",
|
||||
"in Cupertino, California on April 1, 1976. The company's current CEO is Tim Cook, who took \n",
|
||||
"over from Steve Jobs in August 2011. Apple is headquartered at One Apple Park Way in Cupertino.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"# Extract entities\n",
|
||||
"entities = ner_extractor.extract(text)\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(entities)} entities:\")\n",
|
||||
"for entity in entities[:5]:\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', ''))\n",
|
||||
" entity_type = entity.get('type', 'Unknown')\n",
|
||||
" print(f\" - {entity_text} ({entity_type})\")\n"
|
||||
"print(f\" Extracted {len(entities)} entities:\\n\")\n",
|
||||
"print(\"-\" * 80)\n",
|
||||
"\n",
|
||||
"for i, entity in enumerate(entities, 1):\n",
|
||||
" # Handle both dict and object formats\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||||
" \n",
|
||||
" print(f\"{i:2d}. {entity_text:30s} | Type: {entity_type:12s} | Confidence: {confidence:.2f}\")\n",
|
||||
"\n",
|
||||
"print(\"-\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 2: Advanced Entity Recognition\n",
|
||||
"### Understanding Entity Objects\n",
|
||||
"\n",
|
||||
"Use NamedEntityRecognizer for more control.\n"
|
||||
"Each extracted entity contains:\n",
|
||||
"\n",
|
||||
"| Attribute | Description | Example |\n",
|
||||
"|-----------|-------------|----------|\n",
|
||||
"| `text` | The entity text | \"Apple Inc.\" |\n",
|
||||
"| `label/type` | Entity category | \"ORG\" (Organization) |\n",
|
||||
"| `start_char` | Starting position | 0 |\n",
|
||||
"| `end_char` | Ending position | 10 |\n",
|
||||
"| `confidence` | Extraction confidence (0-1) | 0.95 |\n",
|
||||
"| `metadata` | Additional information | {\"method\": \"ml\"} |\n",
|
||||
"\n",
|
||||
"### Common Entity Types\n",
|
||||
"\n",
|
||||
"- **PERSON**: People, including fictional characters\n",
|
||||
"- **ORG**: Companies, agencies, institutions\n",
|
||||
"- **GPE**: Countries, cities, states (Geo-Political Entities)\n",
|
||||
"- **LOC**: Non-GPE locations, mountain ranges, bodies of water\n",
|
||||
"- **DATE**: Absolute or relative dates or periods\n",
|
||||
"- **TIME**: Times smaller than a day\n",
|
||||
"- **MONEY**: Monetary values\n",
|
||||
"- **PERCENT**: Percentage values"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 2: Visualizing Entities in Context\n",
|
||||
"\n",
|
||||
"Let's create a simple visualization to see entities highlighted in the original text."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def highlight_entities(text, entities):\n",
|
||||
" \"\"\"\n",
|
||||
" Create a simple text visualization with entity markers.\n",
|
||||
" \"\"\"\n",
|
||||
" # Group entities by type\n",
|
||||
" entity_types = {}\n",
|
||||
" for entity in entities:\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||||
" \n",
|
||||
" if entity_type not in entity_types:\n",
|
||||
" entity_types[entity_type] = []\n",
|
||||
" entity_types[entity_type].append(entity_text)\n",
|
||||
" \n",
|
||||
" print(\"\\n Entity Visualization:\\n\")\n",
|
||||
" print(\"=\" * 80)\n",
|
||||
" \n",
|
||||
" for entity_type, entity_list in sorted(entity_types.items()):\n",
|
||||
" unique_entities = list(set(entity_list))\n",
|
||||
" print(f\"\\n{entity_type}:\")\n",
|
||||
" for ent in unique_entities:\n",
|
||||
" print(f\" • {ent}\")\n",
|
||||
" \n",
|
||||
" print(\"\\n\" + \"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Visualize the extracted entities\n",
|
||||
"highlight_entities(text, entities)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ️ Step 3: Different Extraction Methods\n",
|
||||
"\n",
|
||||
"Semantica supports multiple extraction methods, each with different strengths:\n",
|
||||
"\n",
|
||||
"### Method Comparison\n",
|
||||
"\n",
|
||||
"| Method | Speed | Accuracy | Use Case | Requires |\n",
|
||||
"|--------|-------|----------|----------|----------|\n",
|
||||
"| **pattern** | | ⭐⭐ | Simple, predictable patterns | Nothing |\n",
|
||||
"| **regex** | | ⭐⭐⭐ | Custom patterns, IDs, codes | Regex knowledge |\n",
|
||||
"| **ml** (spaCy) | | ⭐⭐⭐⭐ | General text, multiple languages | spaCy model |\n",
|
||||
"| **huggingface** | | ⭐⭐⭐⭐⭐ | Domain-specific, fine-tuned | HF model |\n",
|
||||
"| **llm** | | ⭐⭐⭐⭐⭐ | Complex, custom types | API key |\n",
|
||||
"\n",
|
||||
"Let's try different methods:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract.methods import get_entity_method\n",
|
||||
"\n",
|
||||
"sample_text = \"Apple Inc. was founded by Steve Jobs in Cupertino, California in 1976.\"\n",
|
||||
"\n",
|
||||
"print(\" Comparing Extraction Methods:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Try different methods\n",
|
||||
"methods_to_try = [\"pattern\", \"regex\", \"ml\"]\n",
|
||||
"\n",
|
||||
"for method_name in methods_to_try:\n",
|
||||
" try:\n",
|
||||
" print(f\"\\n Method: {method_name.upper()}\")\n",
|
||||
" print(\"-\" * 40)\n",
|
||||
" \n",
|
||||
" method = get_entity_method(method_name)\n",
|
||||
" entities = method(sample_text)\n",
|
||||
" \n",
|
||||
" print(f\"Found {len(entities)} entities:\")\n",
|
||||
" for entity in entities[:5]: # Show first 5\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||||
" print(f\" • {entity_text} ({entity_type})\")\n",
|
||||
" \n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\" ️ Method '{method_name}' not available: {str(e)[:50]}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 4: Advanced Entity Recognition with NamedEntityRecognizer\n",
|
||||
"\n",
|
||||
"`NamedEntityRecognizer` provides more control over the extraction process through configuration parameters.\n",
|
||||
"\n",
|
||||
"### Key Parameters:\n",
|
||||
"\n",
|
||||
"- **`methods`**: List of extraction methods to use (e.g., `[\"spacy\", \"rule-based\"]`)\n",
|
||||
"- **`confidence_threshold`**: Minimum confidence score (0.0-1.0, default: 0.5)\n",
|
||||
"- **`merge_overlapping`**: Whether to merge overlapping entities (default: True)\n",
|
||||
"- **`include_standard_types`**: Include standard entity types (PERSON, ORG, LOC, etc.)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -76,45 +258,384 @@
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NamedEntityRecognizer\n",
|
||||
"\n",
|
||||
"named_entity_recognizer = NamedEntityRecognizer()\n",
|
||||
"# Create recognizer with custom configuration\n",
|
||||
"ner = NamedEntityRecognizer(\n",
|
||||
" methods=[\"spacy\"], # Use spaCy for ML-based extraction\n",
|
||||
" confidence_threshold=0.7, # Only keep high-confidence entities\n",
|
||||
" merge_overlapping=True, # Merge overlapping entity mentions\n",
|
||||
" include_standard_types=True # Include standard entity types\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Sample texts for batch processing\n",
|
||||
"texts = [\n",
|
||||
" \"Tim Cook is the CEO of Apple Inc.\",\n",
|
||||
" \"Microsoft Corporation is headquartered in Redmond, Washington.\",\n",
|
||||
" \"Amazon was founded by Jeff Bezos in 1994.\"\n",
|
||||
" \"Tim Cook is the CEO of Apple Inc., based in Cupertino.\",\n",
|
||||
" \"Microsoft Corporation, founded by Bill Gates, is headquartered in Redmond, Washington.\",\n",
|
||||
" \"Amazon was founded by Jeff Bezos in Seattle in 1994.\",\n",
|
||||
" \"Google was started by Larry Page and Sergey Brin at Stanford University.\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"print(\" Advanced Entity Recognition Results:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"all_entities = []\n",
|
||||
"for text in texts:\n",
|
||||
" entities = named_entity_recognizer.extract_entities(text)\n",
|
||||
"for i, text in enumerate(texts, 1):\n",
|
||||
" entities = ner.extract_entities(text)\n",
|
||||
" all_entities.extend(entities)\n",
|
||||
" print(f\"Text: {text[:40]}...\")\n",
|
||||
" print(f\" Entities: {len(entities)}\")\n",
|
||||
" for entity in entities[:3]:\n",
|
||||
" print(f\" - {entity.get('text', entity.get('entity', ''))} ({entity.get('type', 'Unknown')})\")\n",
|
||||
" print()\n"
|
||||
" \n",
|
||||
" print(f\"\\n Text {i}: {text[:60]}...\")\n",
|
||||
" print(f\" Found {len(entities)} entities:\")\n",
|
||||
" \n",
|
||||
" for entity in entities:\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||||
" print(f\" • {entity_text:25s} | {entity_type:10s} | Confidence: {confidence:.2f}\")\n",
|
||||
"\n",
|
||||
"print(f\"\\n Total entities extracted: {len(all_entities)}\")\n",
|
||||
"print(\"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"## ️ Step 5: Entity Classification\n",
|
||||
"\n",
|
||||
"You've learned how to extract entities from text:\n",
|
||||
"Use `EntityClassifier` to classify and group entities by type, and disambiguate similar entities.\n",
|
||||
"\n",
|
||||
"- **NERExtractor**: Basic entity extraction\n",
|
||||
"- **NamedEntityRecognizer**: Advanced entity recognition with multiple models\n",
|
||||
"### What is EntityClassifier?\n",
|
||||
"\n",
|
||||
"Next: Learn how to extract relationships in the Relation_Extraction notebook.\n"
|
||||
"The `EntityClassifier` helps you:\n",
|
||||
"- **Classify entities** by their type (normalize variations like \"ORG\" vs \"ORGANIZATION\")\n",
|
||||
"- **Group entities** by category for analysis\n",
|
||||
"- **Disambiguate entities** when multiple candidates exist\n",
|
||||
"- **Standardize entity types** across different extraction methods"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import EntityClassifier\n",
|
||||
"\n",
|
||||
"# Initialize classifier\n",
|
||||
"classifier = EntityClassifier()\n",
|
||||
"\n",
|
||||
"# Classify the entities we extracted earlier\n",
|
||||
"classified = classifier.classify_entities(all_entities)\n",
|
||||
"\n",
|
||||
"print(\"️ Entity Classification Results:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"for entity_type, entity_list in sorted(classified.items()):\n",
|
||||
" print(f\"\\n{entity_type} ({len(entity_list)} entities):\")\n",
|
||||
" print(\"-\" * 40)\n",
|
||||
" \n",
|
||||
" # Get unique entity texts\n",
|
||||
" unique_entities = set()\n",
|
||||
" for entity in entity_list:\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||||
" unique_entities.add(entity_text)\n",
|
||||
" \n",
|
||||
" for entity_text in sorted(unique_entities):\n",
|
||||
" print(f\" • {entity_text}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 6: Confidence Scoring\n",
|
||||
"\n",
|
||||
"Use `EntityConfidenceScorer` to assess and improve the confidence scores of extracted entities.\n",
|
||||
"\n",
|
||||
"### Why Confidence Scoring?\n",
|
||||
"\n",
|
||||
"Confidence scores help you:\n",
|
||||
"- **Filter low-quality extractions** (e.g., only keep entities with confidence > 0.8)\n",
|
||||
"- **Prioritize entities** for manual review or validation\n",
|
||||
"- **Assess extraction quality** across different methods or texts\n",
|
||||
"- **Make informed decisions** about which entities to use"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import EntityConfidenceScorer\n",
|
||||
"\n",
|
||||
"# Initialize confidence scorer\n",
|
||||
"scorer = EntityConfidenceScorer()\n",
|
||||
"\n",
|
||||
"# Score the entities\n",
|
||||
"scored_entities = scorer.score_entities(all_entities)\n",
|
||||
"\n",
|
||||
"print(\" Entity Confidence Scoring:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Group by confidence levels\n",
|
||||
"high_confidence = []\n",
|
||||
"medium_confidence = []\n",
|
||||
"low_confidence = []\n",
|
||||
"\n",
|
||||
"for entity in scored_entities:\n",
|
||||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||||
" \n",
|
||||
" if confidence >= 0.8:\n",
|
||||
" high_confidence.append(entity)\n",
|
||||
" elif confidence >= 0.5:\n",
|
||||
" medium_confidence.append(entity)\n",
|
||||
" else:\n",
|
||||
" low_confidence.append(entity)\n",
|
||||
"\n",
|
||||
"print(f\" High Confidence (≥0.8): {len(high_confidence)} entities\")\n",
|
||||
"print(f\"️ Medium Confidence (0.5-0.8): {len(medium_confidence)} entities\")\n",
|
||||
"print(f\" Low Confidence (<0.5): {len(low_confidence)} entities\")\n",
|
||||
"\n",
|
||||
"print(\"\\n Confidence Distribution:\")\n",
|
||||
"print(\"-\" * 40)\n",
|
||||
"\n",
|
||||
"# Show some examples from each category\n",
|
||||
"if high_confidence:\n",
|
||||
" print(\"\\nHigh Confidence Examples:\")\n",
|
||||
" for entity in high_confidence[:3]:\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||||
" confidence = entity.get('confidence', 1.0) if isinstance(entity, dict) else getattr(entity, 'confidence', 1.0)\n",
|
||||
" print(f\" • {entity_text} ({entity_type}) - {confidence:.2f}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 7: Custom Entity Detection\n",
|
||||
"\n",
|
||||
"Use `CustomEntityDetector` to define domain-specific entity patterns.\n",
|
||||
"\n",
|
||||
"### When to Use Custom Patterns?\n",
|
||||
"\n",
|
||||
"Custom patterns are useful for:\n",
|
||||
"- **Domain-specific entities** (e.g., product codes, invoice numbers)\n",
|
||||
"- **Structured identifiers** (e.g., email addresses, phone numbers)\n",
|
||||
"- **Industry-specific terms** (e.g., medical codes, legal citations)\n",
|
||||
"- **Custom formats** not recognized by standard NER"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import CustomEntityDetector\n",
|
||||
"import re\n",
|
||||
"\n",
|
||||
"# Define custom patterns\n",
|
||||
"custom_patterns = {\n",
|
||||
" \"EMAIL\": r'\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b',\n",
|
||||
" \"PHONE\": r'\\b\\d{3}[-.]?\\d{3}[-.]?\\d{4}\\b',\n",
|
||||
" \"PRODUCT_CODE\": r'\\b[A-Z]{2,3}-\\d{4,6}\\b',\n",
|
||||
" \"URL\": r'https?://[^\\s]+'\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# Initialize custom detector\n",
|
||||
"custom_detector = CustomEntityDetector(patterns=custom_patterns)\n",
|
||||
"\n",
|
||||
"# Sample text with custom entities\n",
|
||||
"custom_text = \"\"\"\n",
|
||||
"For support, contact support@apple.com or call 1-800-692-7753.\n",
|
||||
"Order product SKU-12345 from https://store.apple.com.\n",
|
||||
"Technical inquiries: tech@apple.com or visit our website.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"print(\" Custom Entity Detection:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"for entity_type in custom_patterns.keys():\n",
|
||||
" entities = custom_detector.detect_custom_entities(custom_text, entity_type)\n",
|
||||
" \n",
|
||||
" if entities:\n",
|
||||
" print(f\"\\n{entity_type}:\")\n",
|
||||
" for entity in entities:\n",
|
||||
" entity_text = entity.get('text', entity.get('entity', '')) if isinstance(entity, dict) else entity.text\n",
|
||||
" print(f\" • {entity_text}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 8: Batch Processing\n",
|
||||
"\n",
|
||||
"Process multiple documents efficiently using batch processing capabilities.\n",
|
||||
"\n",
|
||||
"### Benefits of Batch Processing:\n",
|
||||
"\n",
|
||||
"- **Performance**: Process multiple documents in one call\n",
|
||||
"- **Consistency**: Same configuration applied to all documents\n",
|
||||
"- **Efficiency**: Reduced overhead from initialization\n",
|
||||
"- **Scalability**: Handle large document collections"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Sample document collection\n",
|
||||
"documents = [\n",
|
||||
" \"Apple Inc. released the iPhone 15 in September 2023.\",\n",
|
||||
" \"Microsoft announced Azure AI updates at Build 2023 in Seattle.\",\n",
|
||||
" \"Google's Sundar Pichai spoke at I/O 2023 in Mountain View, California.\",\n",
|
||||
" \"Tesla's Elon Musk unveiled the Cybertruck in Austin, Texas.\",\n",
|
||||
" \"Amazon Web Services launched new features in Northern Virginia.\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"print(\" Batch Processing Results:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Process all documents\n",
|
||||
"batch_results = ner.process_batch(documents)\n",
|
||||
"\n",
|
||||
"# Analyze results\n",
|
||||
"total_entities = 0\n",
|
||||
"entity_type_counts = {}\n",
|
||||
"\n",
|
||||
"for i, (doc, entities) in enumerate(zip(documents, batch_results), 1):\n",
|
||||
" total_entities += len(entities)\n",
|
||||
" \n",
|
||||
" print(f\"\\n Document {i}:\")\n",
|
||||
" print(f\" Text: {doc[:50]}...\")\n",
|
||||
" print(f\" Entities: {len(entities)}\")\n",
|
||||
" \n",
|
||||
" for entity in entities:\n",
|
||||
" entity_type = entity.get('type', entity.get('label', 'Unknown')) if isinstance(entity, dict) else entity.label\n",
|
||||
" entity_type_counts[entity_type] = entity_type_counts.get(entity_type, 0) + 1\n",
|
||||
"\n",
|
||||
"print(f\"\\n Batch Processing Summary:\")\n",
|
||||
"print(\"-\" * 40)\n",
|
||||
"print(f\"Documents processed: {len(documents)}\")\n",
|
||||
"print(f\"Total entities: {total_entities}\")\n",
|
||||
"print(f\"Average per document: {total_entities/len(documents):.1f}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n Entity Type Distribution:\")\n",
|
||||
"for entity_type, count in sorted(entity_type_counts.items(), key=lambda x: x[1], reverse=True):\n",
|
||||
" print(f\" {entity_type}: {count}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 9: Best Practices & Tips\n",
|
||||
"\n",
|
||||
"### Choosing the Right Method\n",
|
||||
"\n",
|
||||
"1. **Start with ML (spaCy)** for general text\n",
|
||||
"2. **Use patterns/regex** for structured data (IDs, codes)\n",
|
||||
"3. **Try HuggingFace** for domain-specific needs\n",
|
||||
"4. **Consider LLM** for complex, custom entity types\n",
|
||||
"\n",
|
||||
"### ️ Optimizing Performance\n",
|
||||
"\n",
|
||||
"- **Set appropriate confidence thresholds** (0.7-0.8 for production)\n",
|
||||
"- **Use batch processing** for multiple documents\n",
|
||||
"- **Enable merge_overlapping** to reduce duplicates\n",
|
||||
"- **Cache extractors** instead of recreating them\n",
|
||||
"\n",
|
||||
"### Common Pitfalls to Avoid\n",
|
||||
"\n",
|
||||
"- **Don't** use very low confidence thresholds (< 0.5)\n",
|
||||
"- **Don't** process one document at a time in loops\n",
|
||||
"- **Don't** ignore entity metadata (contains useful info)\n",
|
||||
"- **Don't** forget to handle extraction errors\n",
|
||||
"\n",
|
||||
"### When to Use Each Class\n",
|
||||
"\n",
|
||||
"| Use Case | Recommended Class |\n",
|
||||
"|----------|-------------------|\n",
|
||||
"| Quick extraction | `NERExtractor` |\n",
|
||||
"| Fine-tuned control | `NamedEntityRecognizer` |\n",
|
||||
"| Grouping entities | `EntityClassifier` |\n",
|
||||
"| Quality assessment | `EntityConfidenceScorer` |\n",
|
||||
"| Domain-specific | `CustomEntityDetector` |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"\n",
|
||||
"### What You've Learned\n",
|
||||
"\n",
|
||||
"In this notebook, you've learned how to:\n",
|
||||
"\n",
|
||||
" **Extract entities** using `NERExtractor` and `NamedEntityRecognizer` \n",
|
||||
" **Compare different extraction methods** (pattern, regex, ML, HuggingFace, LLM) \n",
|
||||
" **Classify and group entities** with `EntityClassifier` \n",
|
||||
" **Score entity confidence** using `EntityConfidenceScorer` \n",
|
||||
" **Create custom patterns** with `CustomEntityDetector` \n",
|
||||
" **Process documents in batch** for efficiency \n",
|
||||
" **Apply best practices** for production use \n",
|
||||
"\n",
|
||||
"### Key Takeaways\n",
|
||||
"\n",
|
||||
"1. **Multiple methods available**: Choose based on your needs (speed vs accuracy)\n",
|
||||
"2. **Configuration matters**: Tune parameters for optimal results\n",
|
||||
"3. **Confidence is key**: Use thresholds to filter low-quality extractions\n",
|
||||
"4. **Custom patterns work**: For domain-specific entities\n",
|
||||
"5. **Batch processing scales**: Process multiple documents efficiently\n",
|
||||
"\n",
|
||||
"### Next Steps\n",
|
||||
"\n",
|
||||
" **Next Notebook**: [06_Relation_Extraction.ipynb](./06_Relation_Extraction.ipynb) \n",
|
||||
"Learn how to extract relationships between the entities you've identified!\n",
|
||||
"\n",
|
||||
" **Further Reading**:\n",
|
||||
"- [Semantic Extract API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
|
||||
"- [Advanced Extraction Techniques](../advanced/01_Advanced_Extraction.ipynb)\n",
|
||||
"- [Building Knowledge Graphs](./07_Building_Knowledge_Graphs.ipynb)\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
}
|
||||
@@ -4,24 +4,42 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Relation_Extraction.ipynb)\n",
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)\n",
|
||||
"\n",
|
||||
"# Relation Extraction\n",
|
||||
"# Relation Extraction - Comprehensive Guide\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to extract relationships between entities using Semantica's relation extraction modules. You'll learn to use `RelationExtractor` and `TripleExtractor` to identify relationships in text.\n",
|
||||
"This notebook provides a **comprehensive guide** to extracting relationships between entities and building RDF triples using Semantica's relation extraction modules. You'll learn to identify connections, extract structured triples, and prepare data for knowledge graphs.\n",
|
||||
"\n",
|
||||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
|
||||
"\n",
|
||||
"### Learning Objectives\n",
|
||||
"### Learning Objectives\n",
|
||||
"\n",
|
||||
"- Use `RelationExtractor` to extract relationships between entities\n",
|
||||
"- Use `TripleExtractor` to extract RDF triples\n",
|
||||
"- Understand relationship types and confidence scores\n",
|
||||
"- Extract relationships from text with entities\n",
|
||||
"By the end of this notebook, you will be able to:\n",
|
||||
"\n",
|
||||
"## Installation\n",
|
||||
"- Extract relationships using `RelationExtractor`\n",
|
||||
"- Understand different extraction methods (pattern, dependency, co-occurrence, HuggingFace, LLM)\n",
|
||||
"- Configure extraction parameters for optimal results\n",
|
||||
"- Extract RDF triples with `TripleExtractor`\n",
|
||||
"- Validate triples using `TripleValidator`\n",
|
||||
"- Serialize triples to RDF formats with `RDFSerializer`\n",
|
||||
"- Assess triple quality with `TripleQualityChecker`\n",
|
||||
"- Build complete entity → relation → triple pipelines\n",
|
||||
"\n",
|
||||
"### What You'll Learn\n",
|
||||
"\n",
|
||||
"| Component | Purpose | When to Use |\n",
|
||||
"|-----------|---------|-------------|\n",
|
||||
"| `RelationExtractor` | Extract entity relationships | Finding connections |\n",
|
||||
"| `TripleExtractor` | Extract RDF triples | Building knowledge graphs |\n",
|
||||
"| `TripleValidator` | Validate triple quality | Quality assurance |\n",
|
||||
"| `RDFSerializer` | Serialize to RDF formats | Data export |\n",
|
||||
"| `TripleQualityChecker` | Assess triple quality | Quality metrics |\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install Semantica from PyPI:\n",
|
||||
"\n",
|
||||
@@ -31,11 +49,31 @@
|
||||
"pip install semantica[all]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 1: Basic Relation Extraction\n",
|
||||
"\n",
|
||||
"## Step 1: Relation Extraction\n",
|
||||
"Let's start by extracting relationships between entities using `RelationExtractor`.\n",
|
||||
"\n",
|
||||
"Extract relationships using RelationExtractor.\n"
|
||||
"### What is RelationExtractor?\n",
|
||||
"\n",
|
||||
"`RelationExtractor` identifies relationships between entities:\n",
|
||||
"- Finds connections like \"founded_by\", \"located_in\", \"works_for\"\n",
|
||||
"- Returns Relation objects with subject, predicate, object\n",
|
||||
"- Supports multiple extraction methods\n",
|
||||
"- Provides confidence scores for each relation\n",
|
||||
"\n",
|
||||
"### Understanding Relations\n",
|
||||
"\n",
|
||||
"A relation has three parts:\n",
|
||||
"- **Subject**: The source entity (e.g., \"Apple Inc.\")\n",
|
||||
"- **Predicate**: The relationship type (e.g., \"founded_by\")\n",
|
||||
"- **Object**: The target entity (e.g., \"Steve Jobs\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -46,29 +84,268 @@
|
||||
"source": [
|
||||
"from semantica.semantic_extract import RelationExtractor, NERExtractor\n",
|
||||
"\n",
|
||||
"relation_extractor = RelationExtractor()\n",
|
||||
"# Initialize extractors\n",
|
||||
"ner_extractor = NERExtractor()\n",
|
||||
"relation_extractor = RelationExtractor()\n",
|
||||
"\n",
|
||||
"text = \"Tim Cook is the CEO of Apple Inc. Apple Inc. is headquartered in Cupertino, California.\"\n",
|
||||
"# Sample text with clear relationships\n",
|
||||
"text = \"\"\"\n",
|
||||
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
|
||||
"The company is headquartered in Cupertino, California. Tim Cook is the current CEO\n",
|
||||
"of Apple Inc. and took over from Steve Jobs in August 2011.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"# First, extract entities\n",
|
||||
"entities = ner_extractor.extract(text)\n",
|
||||
"print(f\" Extracted {len(entities)} entities\\n\")\n",
|
||||
"\n",
|
||||
"# Then, extract relationships\n",
|
||||
"relationships = relation_extractor.extract(text, entities)\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(entities)} entities and {len(relationships)} relationships\")\n",
|
||||
"for rel in relationships[:5]:\n",
|
||||
" source = rel.get('source', '')\n",
|
||||
" target = rel.get('target', '')\n",
|
||||
" rel_type = rel.get('type', 'related_to')\n",
|
||||
" print(f\" - {source} --[{rel_type}]--> {target}\")\n"
|
||||
"print(f\" Extracted {len(relationships)} relationships:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"for i, rel in enumerate(relationships, 1):\n",
|
||||
" # Handle both dict and object formats\n",
|
||||
" source = rel.get('source', rel.get('subject', '')) if isinstance(rel, dict) else getattr(rel, 'subject', '')\n",
|
||||
" target = rel.get('target', rel.get('object', '')) if isinstance(rel, dict) else getattr(rel, 'object', '')\n",
|
||||
" rel_type = rel.get('type', rel.get('predicate', 'related_to')) if isinstance(rel, dict) else getattr(rel, 'predicate', 'related_to')\n",
|
||||
" confidence = rel.get('confidence', 1.0) if isinstance(rel, dict) else getattr(rel, 'confidence', 1.0)\n",
|
||||
" \n",
|
||||
" # Get source and target text\n",
|
||||
" if isinstance(source, dict):\n",
|
||||
" source_text = source.get('text', source.get('entity', str(source)))\n",
|
||||
" else:\n",
|
||||
" source_text = getattr(source, 'text', str(source))\n",
|
||||
" \n",
|
||||
" if isinstance(target, dict):\n",
|
||||
" target_text = target.get('text', target.get('entity', str(target)))\n",
|
||||
" else:\n",
|
||||
" target_text = getattr(target, 'text', str(target))\n",
|
||||
" \n",
|
||||
" print(f\"{i:2d}. {source_text:20s} --[{rel_type:15s}]--> {target_text:20s} (conf: {confidence:.2f})\")\n",
|
||||
"\n",
|
||||
"print(\"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 2: Triple Extraction\n",
|
||||
"### Understanding Relation Objects\n",
|
||||
"\n",
|
||||
"Extract RDF triples using TripleExtractor.\n"
|
||||
"Each extracted relation contains:\n",
|
||||
"\n",
|
||||
"| Attribute | Description | Example |\n",
|
||||
"|-----------|-------------|----------|\n",
|
||||
"| `subject` | Source entity | Entity(\"Apple Inc.\") |\n",
|
||||
"| `predicate` | Relationship type | \"founded_by\" |\n",
|
||||
"| `object` | Target entity | Entity(\"Steve Jobs\") |\n",
|
||||
"| `confidence` | Extraction confidence (0-1) | 0.85 |\n",
|
||||
"| `context` | Surrounding text | \"Apple Inc. was founded by Steve Jobs\" |\n",
|
||||
"| `metadata` | Additional info | {\"method\": \"pattern\"} |\n",
|
||||
"\n",
|
||||
"### Common Relation Types\n",
|
||||
"\n",
|
||||
"- **founded_by**: Organization founded by person\n",
|
||||
"- **located_in**: Entity located in place\n",
|
||||
"- **works_for**: Person works for organization\n",
|
||||
"- **born_in**: Person born in location\n",
|
||||
"- **part_of**: Entity is part of another\n",
|
||||
"- **related_to**: Generic relationship"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ️ Step 2: Different Extraction Methods\n",
|
||||
"\n",
|
||||
"Semantica supports multiple relation extraction methods:\n",
|
||||
"\n",
|
||||
"### Method Comparison\n",
|
||||
"\n",
|
||||
"| Method | Speed | Accuracy | Use Case | Requires |\n",
|
||||
"|--------|-------|----------|----------|----------|\n",
|
||||
"| **pattern** | | ⭐⭐⭐ | Common relations | Nothing |\n",
|
||||
"| **dependency** | | ⭐⭐⭐⭐ | Grammatical relations | spaCy |\n",
|
||||
"| **cooccurrence** | | ⭐⭐ | Proximity-based | Nothing |\n",
|
||||
"| **huggingface** | | ⭐⭐⭐⭐⭐ | Domain-specific | HF model |\n",
|
||||
"| **llm** | | ⭐⭐⭐⭐⭐ | Complex, custom | API key |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract.methods import get_relation_method\n",
|
||||
"\n",
|
||||
"sample_text = \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n",
|
||||
"sample_entities = ner_extractor.extract(sample_text)\n",
|
||||
"\n",
|
||||
"print(\" Comparing Relation Extraction Methods:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Try different methods\n",
|
||||
"methods_to_try = [\"pattern\", \"dependency\", \"cooccurrence\"]\n",
|
||||
"\n",
|
||||
"for method_name in methods_to_try:\n",
|
||||
" try:\n",
|
||||
" print(f\"\\n Method: {method_name.upper()}\")\n",
|
||||
" print(\"-\" * 40)\n",
|
||||
" \n",
|
||||
" method = get_relation_method(method_name)\n",
|
||||
" relations = method(sample_text, sample_entities)\n",
|
||||
" \n",
|
||||
" print(f\"Found {len(relations)} relations:\")\n",
|
||||
" for rel in relations[:3]: # Show first 3\n",
|
||||
" source = rel.get('source', rel.get('subject', '')) if isinstance(rel, dict) else getattr(rel, 'subject', '')\n",
|
||||
" target = rel.get('target', rel.get('object', '')) if isinstance(rel, dict) else getattr(rel, 'object', '')\n",
|
||||
" rel_type = rel.get('type', rel.get('predicate', 'related_to')) if isinstance(rel, dict) else getattr(rel, 'predicate', 'related_to')\n",
|
||||
" \n",
|
||||
" # Get text representations\n",
|
||||
" source_text = source.get('text', str(source)) if isinstance(source, dict) else getattr(source, 'text', str(source))\n",
|
||||
" target_text = target.get('text', str(target)) if isinstance(target, dict) else getattr(target, 'text', str(target))\n",
|
||||
" \n",
|
||||
" print(f\" • {source_text} --[{rel_type}]--> {target_text}\")\n",
|
||||
" \n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\" ️ Method '{method_name}' not available: {str(e)[:50]}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 3: Advanced Relation Extraction with Configuration\n",
|
||||
"\n",
|
||||
"`RelationExtractor` provides powerful configuration options:\n",
|
||||
"\n",
|
||||
"### Key Parameters:\n",
|
||||
"\n",
|
||||
"- **`relation_types`**: Specific relation types to extract (e.g., `[\"founded\", \"works_at\"]`)\n",
|
||||
"- **`bidirectional`**: Extract bidirectional relations (default: False)\n",
|
||||
"- **`confidence_threshold`**: Minimum confidence score (0.0-1.0, default: 0.6)\n",
|
||||
"- **`max_distance`**: Maximum token distance between entities (default: 50)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create extractor with custom configuration\n",
|
||||
"advanced_extractor = RelationExtractor(\n",
|
||||
" relation_types=[\"founded_by\", \"located_in\", \"works_for\"], # Only extract these types\n",
|
||||
" confidence_threshold=0.7, # Higher threshold for quality\n",
|
||||
" bidirectional=False, # One-way relations only\n",
|
||||
" max_distance=50 # Max 50 tokens between entities\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Sample texts\n",
|
||||
"texts = [\n",
|
||||
" \"Microsoft was founded by Bill Gates and Paul Allen in Albuquerque, New Mexico.\",\n",
|
||||
" \"Satya Nadella works for Microsoft as the CEO.\",\n",
|
||||
" \"Google is located in Mountain View, California.\",\n",
|
||||
" \"Amazon was founded by Jeff Bezos in Seattle, Washington.\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"print(\" Advanced Relation Extraction:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"for i, text in enumerate(texts, 1):\n",
|
||||
" entities = ner_extractor.extract(text)\n",
|
||||
" relations = advanced_extractor.extract(text, entities)\n",
|
||||
" \n",
|
||||
" print(f\"\\n Text {i}: {text}\")\n",
|
||||
" print(f\" Relations found: {len(relations)}\")\n",
|
||||
" \n",
|
||||
" for rel in relations:\n",
|
||||
" source = rel.get('source', rel.get('subject', '')) if isinstance(rel, dict) else getattr(rel, 'subject', '')\n",
|
||||
" target = rel.get('target', rel.get('object', '')) if isinstance(rel, dict) else getattr(rel, 'object', '')\n",
|
||||
" rel_type = rel.get('type', rel.get('predicate', 'related_to')) if isinstance(rel, dict) else getattr(rel, 'predicate', 'related_to')\n",
|
||||
" confidence = rel.get('confidence', 1.0) if isinstance(rel, dict) else getattr(rel, 'confidence', 1.0)\n",
|
||||
" \n",
|
||||
" source_text = source.get('text', str(source)) if isinstance(source, dict) else getattr(source, 'text', str(source))\n",
|
||||
" target_text = target.get('text', str(target)) if isinstance(target, dict) else getattr(target, 'text', str(target))\n",
|
||||
" \n",
|
||||
" print(f\" • {source_text} --[{rel_type}]--> {target_text} (conf: {confidence:.2f})\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ️ Step 4: Relation Classification\n",
|
||||
"\n",
|
||||
"Group and classify extracted relations by their predicate type."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Extract relations from all texts\n",
|
||||
"all_relations = []\n",
|
||||
"for text in texts:\n",
|
||||
" entities = ner_extractor.extract(text)\n",
|
||||
" relations = advanced_extractor.extract(text, entities)\n",
|
||||
" all_relations.extend(relations)\n",
|
||||
"\n",
|
||||
"# Classify relations\n",
|
||||
"classified_relations = advanced_extractor.classify_relations(all_relations)\n",
|
||||
"\n",
|
||||
"print(\"️ Relation Classification:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"for rel_type, rel_list in sorted(classified_relations.items()):\n",
|
||||
" print(f\"\\n{rel_type.upper()} ({len(rel_list)} relations):\")\n",
|
||||
" print(\"-\" * 40)\n",
|
||||
" \n",
|
||||
" for rel in rel_list:\n",
|
||||
" source = rel.get('source', rel.get('subject', '')) if isinstance(rel, dict) else getattr(rel, 'subject', '')\n",
|
||||
" target = rel.get('target', rel.get('object', '')) if isinstance(rel, dict) else getattr(rel, 'object', '')\n",
|
||||
" \n",
|
||||
" source_text = source.get('text', str(source)) if isinstance(source, dict) else getattr(source, 'text', str(source))\n",
|
||||
" target_text = target.get('text', str(target)) if isinstance(target, dict) else getattr(target, 'text', str(target))\n",
|
||||
" \n",
|
||||
" print(f\" • {source_text} → {target_text}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 5: Triple Extraction\n",
|
||||
"\n",
|
||||
"Extract RDF triples using `TripleExtractor`. Triples are the foundation of knowledge graphs.\n",
|
||||
"\n",
|
||||
"### What are RDF Triples?\n",
|
||||
"\n",
|
||||
"RDF (Resource Description Framework) triples are statements with three parts:\n",
|
||||
"- **Subject**: What we're talking about\n",
|
||||
"- **Predicate**: The property or relationship\n",
|
||||
"- **Object**: The value or target\n",
|
||||
"\n",
|
||||
"Example: `(Apple Inc., founded_by, Steve Jobs)`\n",
|
||||
"\n",
|
||||
"### Why Use Triples?\n",
|
||||
"\n",
|
||||
"- **Standardized format** for knowledge representation\n",
|
||||
"- **Compatible** with RDF databases and semantic web\n",
|
||||
"- **Queryable** using SPARQL\n",
|
||||
"- **Interoperable** across systems"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -79,40 +356,338 @@
|
||||
"source": [
|
||||
"from semantica.semantic_extract import TripleExtractor\n",
|
||||
"\n",
|
||||
"triple_extractor = TripleExtractor()\n",
|
||||
"# Initialize triple extractor\n",
|
||||
"triple_extractor = TripleExtractor(\n",
|
||||
" include_temporal=True, # Include temporal information\n",
|
||||
" include_provenance=True # Track source sentences\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"text = \"Apple Inc. was founded by Steve Jobs in 1976. The company is based in Cupertino.\"\n",
|
||||
"# Sample text\n",
|
||||
"triple_text = \"\"\"\n",
|
||||
"Apple Inc. was founded by Steve Jobs in 1976. The company is based in Cupertino, California.\n",
|
||||
"Tim Cook became CEO in 2011. Apple develops the iPhone and MacBook products.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"triples = triple_extractor.extract_triples(text)\n",
|
||||
"# Extract triples\n",
|
||||
"triples = triple_extractor.extract_triples(triple_text)\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(triples)} triples:\")\n",
|
||||
"for triple in triples[:5]:\n",
|
||||
" subject = triple.get('subject', '')\n",
|
||||
" predicate = triple.get('predicate', '')\n",
|
||||
" object_val = triple.get('object', '')\n",
|
||||
" print(f\" - ({subject}, {predicate}, {object_val})\")\n"
|
||||
"print(f\" Extracted {len(triples)} RDF Triples:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"for i, triple in enumerate(triples, 1):\n",
|
||||
" subject = triple.get('subject', '') if isinstance(triple, dict) else triple.subject\n",
|
||||
" predicate = triple.get('predicate', '') if isinstance(triple, dict) else triple.predicate\n",
|
||||
" obj = triple.get('object', '') if isinstance(triple, dict) else triple.object\n",
|
||||
" confidence = triple.get('confidence', 1.0) if isinstance(triple, dict) else getattr(triple, 'confidence', 1.0)\n",
|
||||
" \n",
|
||||
" print(f\"{i:2d}. ({subject}, {predicate}, {obj})\")\n",
|
||||
" print(f\" Confidence: {confidence:.2f}\")\n",
|
||||
" \n",
|
||||
" # Show temporal info if available\n",
|
||||
" metadata = triple.get('metadata', {}) if isinstance(triple, dict) else getattr(triple, 'metadata', {})\n",
|
||||
" if metadata.get('temporal'):\n",
|
||||
" print(f\" Temporal: {metadata['temporal']}\")\n",
|
||||
" print()\n",
|
||||
"\n",
|
||||
"print(\"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"## Step 6: Triple Validation\n",
|
||||
"\n",
|
||||
"You've learned how to extract relationships from text:\n",
|
||||
"Validate extracted triples using `TripleValidator` and assess quality with `TripleQualityChecker`.\n",
|
||||
"\n",
|
||||
"- **RelationExtractor**: Extract relationships between entities\n",
|
||||
"- **TripleExtractor**: Extract RDF triples\n",
|
||||
"### Why Validate Triples?\n",
|
||||
"\n",
|
||||
"Next: Learn how to build knowledge graphs in the Building_Knowledge_Graphs notebook.\n"
|
||||
"- **Ensure completeness**: All parts (subject, predicate, object) present\n",
|
||||
"- **Check confidence**: Filter low-quality extractions\n",
|
||||
"- **Verify consistency**: No contradictory statements\n",
|
||||
"- **Assess quality**: Overall extraction quality metrics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import TripleValidator, TripleQualityChecker\n",
|
||||
"\n",
|
||||
"# Initialize validator and quality checker\n",
|
||||
"validator = TripleValidator()\n",
|
||||
"quality_checker = TripleQualityChecker()\n",
|
||||
"\n",
|
||||
"print(\" Triple Validation:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Validate triples\n",
|
||||
"valid_triples = validator.validate_triples(triples, min_confidence=0.5)\n",
|
||||
"\n",
|
||||
"print(f\"\\n Validation Results:\")\n",
|
||||
"print(f\" Total triples: {len(triples)}\")\n",
|
||||
"print(f\" Valid triples: {len(valid_triples)}\")\n",
|
||||
"print(f\" Filtered out: {len(triples) - len(valid_triples)}\")\n",
|
||||
"\n",
|
||||
"# Check quality\n",
|
||||
"quality_scores = quality_checker.calculate_quality_scores(valid_triples)\n",
|
||||
"\n",
|
||||
"print(f\"\\n Quality Metrics:\")\n",
|
||||
"print(\"-\" * 40)\n",
|
||||
"for metric, value in quality_scores.items():\n",
|
||||
" if isinstance(value, float):\n",
|
||||
" print(f\" {metric}: {value:.2f}\")\n",
|
||||
" else:\n",
|
||||
" print(f\" {metric}: {value}\")\n",
|
||||
"\n",
|
||||
"# Check consistency\n",
|
||||
"consistency_report = validator.check_triple_consistency(valid_triples)\n",
|
||||
"\n",
|
||||
"print(f\"\\n Consistency Check:\")\n",
|
||||
"print(f\" Consistent: {consistency_report.get('consistent', True)}\")\n",
|
||||
"print(f\" Issues found: {len(consistency_report.get('issues', []))}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 7: RDF Serialization\n",
|
||||
"\n",
|
||||
"Serialize triples to various RDF formats using `RDFSerializer`.\n",
|
||||
"\n",
|
||||
"### Supported Formats:\n",
|
||||
"\n",
|
||||
"| Format | Extension | Use Case |\n",
|
||||
"|--------|-----------|----------|\n",
|
||||
"| **Turtle** | .ttl | Human-readable, compact |\n",
|
||||
"| **N-Triples** | .nt | Simple, line-based |\n",
|
||||
"| **JSON-LD** | .jsonld | Web-friendly, JSON-based |\n",
|
||||
"| **RDF/XML** | .rdf | XML-based, verbose |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import RDFSerializer\n",
|
||||
"\n",
|
||||
"# Initialize serializer\n",
|
||||
"serializer = RDFSerializer()\n",
|
||||
"\n",
|
||||
"print(\" RDF Serialization Examples:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Serialize to different formats\n",
|
||||
"formats = [\"turtle\", \"ntriples\", \"jsonld\"]\n",
|
||||
"\n",
|
||||
"for fmt in formats:\n",
|
||||
" print(f\"\\n {fmt.upper()} Format:\")\n",
|
||||
" print(\"-\" * 40)\n",
|
||||
" \n",
|
||||
" try:\n",
|
||||
" serialized = serializer.serialize_to_rdf(valid_triples[:3], format=fmt) # Show first 3\n",
|
||||
" \n",
|
||||
" # Show preview (first 300 chars)\n",
|
||||
" preview = serialized[:300] + \"...\" if len(serialized) > 300 else serialized\n",
|
||||
" print(preview)\n",
|
||||
" \n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\"Error: {str(e)[:50]}\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 8: Complete Extraction Pipeline\n",
|
||||
"\n",
|
||||
"Let's build a complete pipeline: **Entities → Relations → Triples**\n",
|
||||
"\n",
|
||||
"This demonstrates the full workflow for knowledge graph construction."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def extract_knowledge(text):\n",
|
||||
" \"\"\"\n",
|
||||
" Complete knowledge extraction pipeline.\n",
|
||||
" \n",
|
||||
" Args:\n",
|
||||
" text: Input text\n",
|
||||
" \n",
|
||||
" Returns:\n",
|
||||
" dict: Extracted entities, relations, and triples\n",
|
||||
" \"\"\"\n",
|
||||
" # Step 1: Extract entities\n",
|
||||
" entities = ner_extractor.extract(text)\n",
|
||||
" \n",
|
||||
" # Step 2: Extract relations\n",
|
||||
" relations = relation_extractor.extract(text, entities)\n",
|
||||
" \n",
|
||||
" # Step 3: Extract triples\n",
|
||||
" triples = triple_extractor.extract_triples(text, entities=entities, relationships=relations)\n",
|
||||
" \n",
|
||||
" # Step 4: Validate triples\n",
|
||||
" valid_triples = validator.validate_triples(triples)\n",
|
||||
" \n",
|
||||
" return {\n",
|
||||
" 'entities': entities,\n",
|
||||
" 'relations': relations,\n",
|
||||
" 'triples': valid_triples\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
"# Sample knowledge-rich text\n",
|
||||
"knowledge_text = \"\"\"\n",
|
||||
"Tesla Inc. was founded by Elon Musk, JB Straubel, Martin Eberhard, Marc Tarpenning, \n",
|
||||
"and Ian Wright in 2003. The company is headquartered in Austin, Texas. Tesla produces \n",
|
||||
"electric vehicles including the Model S, Model 3, Model X, and Model Y. Elon Musk serves \n",
|
||||
"as CEO and has been instrumental in the company's growth.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"print(\" Complete Extraction Pipeline:\\n\")\n",
|
||||
"print(\"=\" * 80)\n",
|
||||
"\n",
|
||||
"# Run pipeline\n",
|
||||
"result = extract_knowledge(knowledge_text)\n",
|
||||
"\n",
|
||||
"print(f\"\\n Extraction Results:\")\n",
|
||||
"print(\"-\" * 40)\n",
|
||||
"print(f\"Entities extracted: {len(result['entities'])}\")\n",
|
||||
"print(f\"Relations extracted: {len(result['relations'])}\")\n",
|
||||
"print(f\"Triples extracted: {len(result['triples'])}\")\n",
|
||||
"\n",
|
||||
"print(f\"\\n Sample Triples:\")\n",
|
||||
"for i, triple in enumerate(result['triples'][:5], 1):\n",
|
||||
" subject = triple.get('subject', '') if isinstance(triple, dict) else triple.subject\n",
|
||||
" predicate = triple.get('predicate', '') if isinstance(triple, dict) else triple.predicate\n",
|
||||
" obj = triple.get('object', '') if isinstance(triple, dict) else triple.object\n",
|
||||
" print(f\" {i}. ({subject}, {predicate}, {obj})\")\n",
|
||||
"\n",
|
||||
"print(\"\\n\" + \"=\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 9: Best Practices & Tips\n",
|
||||
"\n",
|
||||
"### Choosing the Right Method\n",
|
||||
"\n",
|
||||
"1. **Start with pattern-based** for common relations\n",
|
||||
"2. **Use dependency parsing** for grammatical accuracy\n",
|
||||
"3. **Try co-occurrence** for exploratory analysis\n",
|
||||
"4. **Consider LLM** for complex, domain-specific relations\n",
|
||||
"\n",
|
||||
"### ️ Optimizing Extraction\n",
|
||||
"\n",
|
||||
"- **Set confidence thresholds** (0.6-0.7 for production)\n",
|
||||
"- **Specify relation_types** to focus extraction\n",
|
||||
"- **Adjust max_distance** based on text structure\n",
|
||||
"- **Validate triples** before using in knowledge graphs\n",
|
||||
"\n",
|
||||
"### Common Pitfalls to Avoid\n",
|
||||
"\n",
|
||||
"- **Don't** skip entity extraction (relations need entities)\n",
|
||||
"- **Don't** use very low confidence thresholds\n",
|
||||
"- **Don't** ignore relation validation\n",
|
||||
"- **Don't** forget to serialize triples for storage\n",
|
||||
"\n",
|
||||
"### When to Use Each Component\n",
|
||||
"\n",
|
||||
"| Use Case | Recommended Component |\n",
|
||||
"|----------|----------------------|\n",
|
||||
"| Find entity connections | `RelationExtractor` |\n",
|
||||
"| Build knowledge graphs | `TripleExtractor` |\n",
|
||||
"| Quality assurance | `TripleValidator` |\n",
|
||||
"| Export to RDF | `RDFSerializer` |\n",
|
||||
"| Assess extraction quality | `TripleQualityChecker` |\n",
|
||||
"\n",
|
||||
"### Performance Tips\n",
|
||||
"\n",
|
||||
"1. **Extract entities once**, reuse for relations and triples\n",
|
||||
"2. **Batch process** multiple documents together\n",
|
||||
"3. **Cache extractors** instead of recreating\n",
|
||||
"4. **Filter early** with confidence thresholds\n",
|
||||
"5. **Validate incrementally** rather than all at once"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Summary\n",
|
||||
"\n",
|
||||
"### What You've Learned\n",
|
||||
"\n",
|
||||
"In this notebook, you've learned how to:\n",
|
||||
"\n",
|
||||
" **Extract relationships** using `RelationExtractor` \n",
|
||||
" **Compare extraction methods** (pattern, dependency, co-occurrence, HuggingFace, LLM) \n",
|
||||
" **Configure extraction parameters** for optimal results \n",
|
||||
" **Extract RDF triples** with `TripleExtractor` \n",
|
||||
" **Validate triples** using `TripleValidator` \n",
|
||||
" **Serialize to RDF formats** with `RDFSerializer` \n",
|
||||
" **Assess quality** with `TripleQualityChecker` \n",
|
||||
" **Build complete pipelines** from entities to triples \n",
|
||||
"\n",
|
||||
"### Key Takeaways\n",
|
||||
"\n",
|
||||
"1. **Relations connect entities**: They form the backbone of knowledge graphs\n",
|
||||
"2. **Multiple methods available**: Choose based on accuracy vs speed needs\n",
|
||||
"3. **Configuration is powerful**: Tune parameters for your domain\n",
|
||||
"4. **Triples are standardized**: Use RDF for interoperability\n",
|
||||
"5. **Validation is essential**: Ensure quality before using triples\n",
|
||||
"6. **Pipelines are efficient**: Extract entities → relations → triples in sequence\n",
|
||||
"\n",
|
||||
"### Next Steps\n",
|
||||
"\n",
|
||||
" **Next Notebook**: [07_Building_Knowledge_Graphs.ipynb](./07_Building_Knowledge_Graphs.ipynb) \n",
|
||||
"Learn how to build complete knowledge graphs from your extracted triples!\n",
|
||||
"\n",
|
||||
" **Further Reading**:\n",
|
||||
"- [Semantic Extract API Reference](https://semantica.readthedocs.io/reference/semantic_extract/)\n",
|
||||
"- [Knowledge Graph Module](https://semantica.readthedocs.io/reference/kg/)\n",
|
||||
"- [Advanced Graph Analytics](../advanced/02_Advanced_Graph_Analytics.ipynb)\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"**Questions or Issues?** Check out our [GitHub repository](https://github.com/Hawksight-AI/semantica) or [documentation](https://semantica.readthedocs.io)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
}
|
||||
@@ -20,7 +20,7 @@
|
||||
"- **Multi-hop Reasoning**: Follows relationships across the graph for deeper context\n",
|
||||
"- **20+ Semantica Modules**: Demonstrates comprehensive use of the framework\n",
|
||||
"\n",
|
||||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/) \u2022 [GraphRAG Guide](https://semantica.readthedocs.io/concepts/)\n",
|
||||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/) • [GraphRAG Guide](https://semantica.readthedocs.io/concepts/)\n",
|
||||
"\n",
|
||||
"### What You'll Learn\n",
|
||||
"\n",
|
||||
@@ -33,7 +33,7 @@
|
||||
"\n",
|
||||
"### Pipeline Overview\n",
|
||||
"\n",
|
||||
"**Real-World Data Sources (MCP/Web/Feeds) \u2192 Parse \u2192 Extract Entities & Relationships \u2192 Build Knowledge Graph \u2192 Generate Embeddings \u2192 Vector Store \u2192 Hybrid Search \u2192 Context Retrieval \u2192 GraphRAG Query System \u2192 LLM Integration \u2192 Answer Generation**\n",
|
||||
"**Real-World Data Sources (MCP/Web/Feeds) → Parse → Extract Entities & Relationships → Build Knowledge Graph → Generate Embeddings → Vector Store → Hybrid Search → Context Retrieval → GraphRAG Query System → LLM Integration → Answer Generation**\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
@@ -320,122 +320,33 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
|
||||
"from semantica.split import (\n",
|
||||
" SemanticChunker, EntityAwareChunker, RelationAwareChunker, GraphBasedChunker\n",
|
||||
")\n",
|
||||
"import numpy as np\n",
|
||||
"from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripleExtractor\n",
|
||||
"\n",
|
||||
"print(\"Step 1: Creating chunks for Vector Store (semantic chunking)...\")\n",
|
||||
"semantic_chunker = SemanticChunker(\n",
|
||||
" chunk_size=1000,\n",
|
||||
" chunk_overlap=200\n",
|
||||
")\n",
|
||||
"print(\"Extracting entities, relationships, and triples...\")\n",
|
||||
"\n",
|
||||
"vector_store_chunks = []\n",
|
||||
"for i, doc in enumerate(parsed_documents):\n",
|
||||
" doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n",
|
||||
" if doc_text.strip():\n",
|
||||
" chunks = semantic_chunker.chunk(doc_text)\n",
|
||||
" if isinstance(chunks, list):\n",
|
||||
" for chunk in chunks:\n",
|
||||
" if hasattr(chunk, 'metadata'):\n",
|
||||
" chunk.metadata['chunking_method'] = 'semantic'\n",
|
||||
" chunk.metadata['store_type'] = 'vector'\n",
|
||||
" chunk.metadata['source_doc'] = i\n",
|
||||
" vector_store_chunks.extend(chunks)\n",
|
||||
" else:\n",
|
||||
" if hasattr(chunks, 'metadata'):\n",
|
||||
" chunks.metadata['chunking_method'] = 'semantic'\n",
|
||||
" chunks.metadata['store_type'] = 'vector'\n",
|
||||
" chunks.metadata['source_doc'] = i\n",
|
||||
" vector_store_chunks.append(chunks)\n",
|
||||
"ner = NamedEntityRecognizer()\n",
|
||||
"rel_extractor = RelationExtractor()\n",
|
||||
"triple_extractor = TripleExtractor()\n",
|
||||
"\n",
|
||||
"print(f\"Created {len(vector_store_chunks)} semantic chunks for vector store\")\n",
|
||||
"flat_entities = []\n",
|
||||
"flat_relationships = []\n",
|
||||
"flat_triples = []\n",
|
||||
"\n",
|
||||
"print(\"\\nStep 2: Extracting entities/relationships for graph-aware chunking...\")\n",
|
||||
"ner_extractor = NERExtractor()\n",
|
||||
"relation_extractor = RelationExtractor()\n",
|
||||
"\n",
|
||||
"doc_entities = {}\n",
|
||||
"doc_relationships = {}\n",
|
||||
"\n",
|
||||
"for i, doc in enumerate(parsed_documents):\n",
|
||||
" doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n",
|
||||
" if doc_text.strip():\n",
|
||||
" entities = ner_extractor.extract(doc_text)\n",
|
||||
" if isinstance(entities, list):\n",
|
||||
" doc_entities[i] = entities\n",
|
||||
" else:\n",
|
||||
" doc_entities[i] = [entities] if entities else []\n",
|
||||
" \n",
|
||||
" relationships = relation_extractor.extract(doc_text, doc_entities[i])\n",
|
||||
" if isinstance(relationships, list):\n",
|
||||
" doc_relationships[i] = relationships\n",
|
||||
" else:\n",
|
||||
" doc_relationships[i] = [relationships] if relationships else []\n",
|
||||
"\n",
|
||||
"print(f\"Extracted entities from {len(doc_entities)} documents\")\n",
|
||||
"print(f\"Extracted relationships from {len(doc_relationships)} documents\")\n",
|
||||
"\n",
|
||||
"print(\"\\nStep 3: Creating chunks for Graph Store (graph-aware chunking)...\")\n",
|
||||
"entity_chunker = EntityAwareChunker(\n",
|
||||
" chunk_size=1000,\n",
|
||||
" chunk_overlap=200,\n",
|
||||
" ner_method=\"spacy\",\n",
|
||||
" preserve_entities=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"relation_chunker = RelationAwareChunker(\n",
|
||||
" chunk_size=1000,\n",
|
||||
" chunk_overlap=200,\n",
|
||||
" preserve_triples=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"graph_store_chunks = []\n",
|
||||
"\n",
|
||||
"for i, doc in enumerate(parsed_documents):\n",
|
||||
" doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n",
|
||||
" if not doc_text.strip():\n",
|
||||
" continue\n",
|
||||
"for doc in normalized_documents:\n",
|
||||
" text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n",
|
||||
" \n",
|
||||
" if i in doc_relationships and len(doc_relationships[i]) > 0:\n",
|
||||
" chunks = relation_chunker.chunk(\n",
|
||||
" doc_text,\n",
|
||||
" relationships=doc_relationships[i]\n",
|
||||
" )\n",
|
||||
" elif i in doc_entities and len(doc_entities[i]) > 0:\n",
|
||||
" chunks = entity_chunker.chunk(\n",
|
||||
" doc_text,\n",
|
||||
" entities=doc_entities[i]\n",
|
||||
" )\n",
|
||||
" else:\n",
|
||||
" chunks = entity_chunker.chunk(doc_text)\n",
|
||||
" entities = ner.extract_entities(text)\n",
|
||||
" flat_entities.extend(entities if isinstance(entities, list) else [entities])\n",
|
||||
" \n",
|
||||
" if isinstance(chunks, list):\n",
|
||||
" for chunk in chunks:\n",
|
||||
" if hasattr(chunk, 'metadata'):\n",
|
||||
" chunk.metadata['chunking_method'] = 'graph_aware'\n",
|
||||
" chunk.metadata['store_type'] = 'graph'\n",
|
||||
" chunk.metadata['source_doc'] = i\n",
|
||||
" if i in doc_entities:\n",
|
||||
" chunk.metadata['entities'] = doc_entities[i]\n",
|
||||
" if i in doc_relationships:\n",
|
||||
" chunk.metadata['relationships'] = doc_relationships[i]\n",
|
||||
" graph_store_chunks.extend(chunks)\n",
|
||||
" else:\n",
|
||||
" if hasattr(chunks, 'metadata'):\n",
|
||||
" chunks.metadata['chunking_method'] = 'graph_aware'\n",
|
||||
" chunks.metadata['store_type'] = 'graph'\n",
|
||||
" chunks.metadata['source_doc'] = i\n",
|
||||
" graph_store_chunks.append(chunks)\n",
|
||||
" relations = rel_extractor.extract_relations(text, entities=entities)\n",
|
||||
" flat_relationships.extend(relations if isinstance(relations, list) else [relations])\n",
|
||||
" \n",
|
||||
" triples = triple_extractor.extract_triples(text, entities=entities, relationships=relations)\n",
|
||||
" flat_triples.extend(triples if isinstance(triples, list) else [triples])\n",
|
||||
"\n",
|
||||
"print(f\"Created {len(graph_store_chunks)} graph-aware chunks for graph store\")\n",
|
||||
"\n",
|
||||
"chunked_documents = vector_store_chunks + graph_store_chunks\n",
|
||||
"print(f\"\\nTotal chunks: {len(chunked_documents)}\")\n",
|
||||
"print(f\" Vector store chunks: {len(vector_store_chunks)}\")\n",
|
||||
"print(f\" Graph store chunks: {len(graph_store_chunks)}\")\n"
|
||||
"print(f\"Extracted {len(flat_entities)} entities\")\n",
|
||||
"print(f\"Extracted {len(flat_relationships)} relationships\")\n",
|
||||
"print(f\"Extracted {len(flat_triples)} triples\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -600,8 +511,8 @@
|
||||
"resolved_result = resolve_entities(deduplicated_entities, method=\"fuzzy\")\n",
|
||||
"resolved_entities = resolved_result.get('entities', deduplicated_entities)\n",
|
||||
"\n",
|
||||
"print(f\"Deduplicated: {len(flat_entities)} \u2192 {len(deduplicated_entities)} entities\")\n",
|
||||
"print(f\"Resolved: {len(deduplicated_entities)} \u2192 {len(resolved_entities)} entities\")\n",
|
||||
"print(f\"Deduplicated: {len(flat_entities)} → {len(deduplicated_entities)} entities\")\n",
|
||||
"print(f\"Resolved: {len(deduplicated_entities)} → {len(resolved_entities)} entities\")\n",
|
||||
"\n",
|
||||
"print(\"Building knowledge graph...\")\n",
|
||||
"\n",
|
||||
@@ -1545,9 +1456,9 @@
|
||||
" \n",
|
||||
" print(f\"\\nComplete workflow executed successfully!\")\n",
|
||||
" print(f\"Final Results:\")\n",
|
||||
" print(f\" Query processed: \u2713\")\n",
|
||||
" print(f\" Query processed: ✓\")\n",
|
||||
" print(f\" Context retrieved: {workflow_result['metrics']['context_items']} items\")\n",
|
||||
" print(f\" Answer generated: \u2713\")\n",
|
||||
" print(f\" Answer generated: ✓\")\n",
|
||||
"else:\n",
|
||||
" print(\"Configure data sources above to run complete workflow with real data\")\n"
|
||||
]
|
||||
@@ -1587,11 +1498,11 @@
|
||||
" print(f\"{feature:<20} {trad:<25} {graph:<25}\")\n",
|
||||
"\n",
|
||||
"print(\"\\nGraphRAG Advantages:\")\n",
|
||||
"print(f\" \u2022 Better handling of complex queries requiring relationship understanding\")\n",
|
||||
"print(f\" \u2022 Multi-hop reasoning across entities\")\n",
|
||||
"print(f\" \u2022 More accurate answers through structured knowledge\")\n",
|
||||
"print(f\" \u2022 Better explainability with graph paths\")\n",
|
||||
"print(f\" \u2022 Reduced hallucinations through graph validation\")\n"
|
||||
"print(f\" • Better handling of complex queries requiring relationship understanding\")\n",
|
||||
"print(f\" • Multi-hop reasoning across entities\")\n",
|
||||
"print(f\" • More accurate answers through structured knowledge\")\n",
|
||||
"print(f\" • Better explainability with graph paths\")\n",
|
||||
"print(f\" • Reduced hallucinations through graph validation\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1651,9 +1562,9 @@
|
||||
"\n",
|
||||
"# Save vector store (if supported)\n",
|
||||
"print(\"\\nVector Store:\")\n",
|
||||
"print(f\" Vectors stored: \u2713\")\n",
|
||||
"print(f\" Metadata stored: \u2713\")\n",
|
||||
"print(f\" Ready for reuse: \u2713\")\n"
|
||||
"print(f\" Vectors stored: ✓\")\n",
|
||||
"print(f\" Metadata stored: ✓\")\n",
|
||||
"print(f\" Ready for reuse: ✓\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -83,7 +83,6 @@
|
||||
### NamedEntityRecognizer
|
||||
|
||||
Coordinator for entity extraction.
|
||||
<<<<<<< HEAD
|
||||
|
||||
**Parameters:**
|
||||
|
||||
@@ -93,8 +92,6 @@ Coordinator for entity extraction.
|
||||
| `confidence_threshold` | float | `0.5` | Minimum confidence score |
|
||||
| `merge_overlapping` | bool | `True` | Merge overlapping entities |
|
||||
| `include_standard_types` | bool | `True` | Include Person, Org, Location |
|
||||
=======
|
||||
>>>>>>> origin/main
|
||||
|
||||
**Methods:**
|
||||
|
||||
@@ -108,7 +105,6 @@ Coordinator for entity extraction.
|
||||
```python
|
||||
from semantica.semantic_extract import NamedEntityRecognizer
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Basic usage
|
||||
ner = NamedEntityRecognizer()
|
||||
entities = ner.extract_entities("Elon Musk leads SpaceX.")
|
||||
@@ -121,17 +117,11 @@ ner = NamedEntityRecognizer(
|
||||
merge_overlapping=True
|
||||
)
|
||||
entities = ner.extract_entities("Apple Inc. was founded in 1976.")
|
||||
=======
|
||||
ner = NamedEntityRecognizer()
|
||||
entities = ner.extract_entities("Elon Musk leads SpaceX.")
|
||||
# [Entity(text="Elon Musk", label="PERSON"), Entity(text="SpaceX", label="ORG")]
|
||||
>>>>>>> origin/main
|
||||
```
|
||||
|
||||
### RelationExtractor
|
||||
|
||||
Extracts relationships between entities.
|
||||
<<<<<<< HEAD
|
||||
|
||||
**Parameters:**
|
||||
|
||||
@@ -141,8 +131,6 @@ Extracts relationships between entities.
|
||||
| `bidirectional` | bool | `False` | Extract bidirectional relations |
|
||||
| `confidence_threshold` | float | `0.6` | Minimum confidence score |
|
||||
| `max_distance` | int | `50` | Max token distance between entities |
|
||||
=======
|
||||
>>>>>>> origin/main
|
||||
|
||||
**Methods:**
|
||||
|
||||
@@ -155,7 +143,6 @@ Extracts relationships between entities.
|
||||
```python
|
||||
from semantica.semantic_extract import RelationExtractor, NamedEntityRecognizer
|
||||
|
||||
<<<<<<< HEAD
|
||||
# First extract entities
|
||||
ner = NamedEntityRecognizer()
|
||||
text = "Elon Musk founded SpaceX in 2002."
|
||||
@@ -173,16 +160,10 @@ rel_extractor = RelationExtractor(
|
||||
bidirectional=False
|
||||
)
|
||||
relations = rel_extractor.extract_relations(text, entities=entities)
|
||||
=======
|
||||
re = RelationExtractor()
|
||||
relations = re.extract_relations(text, entities)
|
||||
# [Relation(source="Elon Musk", target="SpaceX", type="leads")]
|
||||
>>>>>>> origin/main
|
||||
```
|
||||
|
||||
### EventDetector
|
||||
|
||||
<<<<<<< HEAD
|
||||
Identifies events with temporal information and participants.
|
||||
|
||||
**Parameters:**
|
||||
@@ -193,9 +174,6 @@ Identifies events with temporal information and participants.
|
||||
| `extract_participants` | bool | `True` | Extract event participants |
|
||||
| `extract_location` | bool | `True` | Extract event locations |
|
||||
| `extract_time` | bool | `True` | Extract temporal information |
|
||||
=======
|
||||
Identifies events.
|
||||
>>>>>>> origin/main
|
||||
|
||||
**Methods:**
|
||||
|
||||
@@ -203,7 +181,6 @@ Identifies events.
|
||||
|--------|-------------|
|
||||
| `detect_events(text)` | Find events |
|
||||
|
||||
<<<<<<< HEAD
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
@@ -227,18 +204,12 @@ Extracts RDF triples (Subject-Predicate-Object).
|
||||
|-----------|------|---------|-------------|
|
||||
| `include_temporal` | bool | `False` | Include time information |
|
||||
| `include_provenance` | bool | `False` | Track source sentences |
|
||||
=======
|
||||
### TripleExtractor
|
||||
|
||||
Extracts RDF triples.
|
||||
>>>>>>> origin/main
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `extract_triples(text)` | Get (S, P, O) tuples |
|
||||
<<<<<<< HEAD
|
||||
|
||||
**Example:**
|
||||
|
||||
@@ -252,15 +223,12 @@ extractor = TripleExtractor(
|
||||
triples = extractor.extract_triples("Steve Jobs founded Apple in 1976.")
|
||||
# [Triple(subject="Steve Jobs", predicate="founded", object="Apple", temporal="1976")]
|
||||
```
|
||||
=======
|
||||
>>>>>>> origin/main
|
||||
|
||||
---
|
||||
|
||||
## Convenience Functions
|
||||
## Usage Examples
|
||||
|
||||
```python
|
||||
<<<<<<< HEAD
|
||||
from semantica.semantic_extract import (
|
||||
NamedEntityRecognizer,
|
||||
RelationExtractor,
|
||||
@@ -295,19 +263,6 @@ print(f"Entities: {len(entities)}")
|
||||
print(f"Relations: {len(relations)}")
|
||||
print(f"Triples: {len(triples)}")
|
||||
print(f"Events: {len(events)}")
|
||||
=======
|
||||
from semantica.semantic_extract import build
|
||||
|
||||
# All-in-one extraction
|
||||
result = build(
|
||||
"Apple released the iPhone in 2007.",
|
||||
extract_entities=True,
|
||||
extract_relations=True,
|
||||
extract_events=True
|
||||
)
|
||||
|
||||
print(result['triples'])
|
||||
>>>>>>> origin/main
|
||||
```
|
||||
|
||||
---
|
||||
@@ -344,7 +299,6 @@ semantic_extract:
|
||||
### KG Population Pipeline
|
||||
|
||||
```python
|
||||
<<<<<<< HEAD
|
||||
from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripleExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
@@ -362,23 +316,6 @@ sources = [{
|
||||
"relationships": [{"source": t.subject, "target": t.object, "type": t.predicate} for t in triples]
|
||||
}]
|
||||
kg = builder.build(sources)
|
||||
=======
|
||||
from semantica.semantic_extract import build
|
||||
from semantica.kg import KnowledgeGraph
|
||||
|
||||
# 1. Extract
|
||||
text = "Google was founded by Larry Page and Sergey Brin."
|
||||
data = build(text, extract_triples=True)
|
||||
|
||||
# 2. Populate KG
|
||||
kg = KnowledgeGraph()
|
||||
for triple in data['triples']:
|
||||
kg.add_triple(
|
||||
subject=triple.subject,
|
||||
predicate=triple.predicate,
|
||||
object=triple.object
|
||||
)
|
||||
>>>>>>> origin/main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -183,154 +183,5 @@ __all__ = [
|
||||
"get_entity_method",
|
||||
"get_relation_method",
|
||||
"get_triple_method",
|
||||
# Convenience
|
||||
"build",
|
||||
]
|
||||
|
||||
|
||||
def build(
|
||||
text: Union[str, List[str]],
|
||||
extract_entities: bool = True,
|
||||
extract_relations: bool = True,
|
||||
extract_events: bool = False,
|
||||
extract_triples: bool = False,
|
||||
resolve_coreferences: bool = False,
|
||||
**options,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract semantic information from text (module-level convenience function).
|
||||
|
||||
This is a user-friendly wrapper that performs comprehensive semantic extraction
|
||||
including entities, relations, events, and triples.
|
||||
|
||||
Args:
|
||||
text: Input text or list of texts to process
|
||||
extract_entities: Whether to extract named entities (default: True)
|
||||
extract_relations: Whether to extract relationships (default: True)
|
||||
extract_events: Whether to extract events (default: False)
|
||||
extract_triples: Whether to extract RDF triples (default: False)
|
||||
resolve_coreferences: Whether to resolve coreferences (default: False)
|
||||
**options: Additional extraction options
|
||||
|
||||
Returns:
|
||||
Dictionary containing:
|
||||
- entities: List of extracted entities
|
||||
- relations: List of extracted relationships
|
||||
- events: List of extracted events (if enabled)
|
||||
- triples: List of extracted triples (if enabled)
|
||||
- coreferences: Coreference resolution results (if enabled)
|
||||
- metadata: Extraction metadata
|
||||
- statistics: Extraction statistics
|
||||
|
||||
Examples:
|
||||
>>> import semantica
|
||||
>>> result = semantica.semantic_extract.build(
|
||||
... text="Apple Inc. was founded by Steve Jobs in 1976.",
|
||||
... extract_entities=True,
|
||||
... extract_relations=True
|
||||
... )
|
||||
>>> print(f"Extracted {len(result['entities'])} entities")
|
||||
"""
|
||||
# Normalize text to list
|
||||
is_single = isinstance(text, str)
|
||||
if is_single:
|
||||
texts = [text]
|
||||
else:
|
||||
texts = text
|
||||
|
||||
results = {
|
||||
"entities": [],
|
||||
"relations": [],
|
||||
"events": [],
|
||||
"triples": [],
|
||||
"coreferences": [],
|
||||
"metadata": {},
|
||||
"statistics": {},
|
||||
}
|
||||
|
||||
# Initialize extractors
|
||||
if extract_entities:
|
||||
ner = NamedEntityRecognizer(config=options.get("ner_config", {}), **options)
|
||||
|
||||
if extract_relations:
|
||||
from .relation_extractor import RelationExtractor
|
||||
|
||||
rel_extractor = RelationExtractor(
|
||||
**options.get("relation_config", {}), **options
|
||||
)
|
||||
|
||||
if extract_events:
|
||||
from .event_detector import EventDetector
|
||||
|
||||
event_detector = EventDetector(**options.get("event_config", {}), **options)
|
||||
|
||||
if extract_triples:
|
||||
from .triple_extractor import TripleExtractor
|
||||
|
||||
triple_extractor = TripleExtractor(
|
||||
**options.get("triple_config", {}), **options
|
||||
)
|
||||
|
||||
if resolve_coreferences:
|
||||
from .coreference_resolver import CoreferenceResolver
|
||||
|
||||
coref_resolver = CoreferenceResolver(
|
||||
**options.get("coref_config", {}), **options
|
||||
)
|
||||
|
||||
# Process texts
|
||||
all_entities = []
|
||||
all_relations = []
|
||||
all_events = []
|
||||
all_triples = []
|
||||
|
||||
for txt in texts:
|
||||
if extract_entities:
|
||||
entities = ner.extract_entities(txt, **options)
|
||||
all_entities.extend(entities)
|
||||
|
||||
if extract_relations:
|
||||
# Relations typically need entities, so extract if not already done
|
||||
if not extract_entities:
|
||||
entities = (
|
||||
ner.extract_entities(txt, **options) if "ner" in locals() else []
|
||||
)
|
||||
relations = rel_extractor.extract_relations(
|
||||
txt, entities=entities if extract_entities else [], **options
|
||||
)
|
||||
all_relations.extend(relations)
|
||||
|
||||
if extract_events:
|
||||
events = event_detector.detect_events(txt, **options)
|
||||
all_events.extend(events)
|
||||
|
||||
if extract_triples:
|
||||
triples = triple_extractor.extract_triples(txt, **options)
|
||||
all_triples.extend(triples)
|
||||
|
||||
if resolve_coreferences:
|
||||
corefs = coref_resolver.resolve(txt, **options)
|
||||
results["coreferences"].append(corefs)
|
||||
|
||||
results["entities"] = all_entities
|
||||
results["relations"] = all_relations
|
||||
results["events"] = all_events
|
||||
results["triples"] = all_triples
|
||||
|
||||
results["statistics"] = {
|
||||
"texts_processed": len(texts),
|
||||
"entities_extracted": len(all_entities),
|
||||
"relations_extracted": len(all_relations),
|
||||
"events_extracted": len(all_events),
|
||||
"triples_extracted": len(all_triples),
|
||||
}
|
||||
|
||||
results["metadata"] = {
|
||||
"extract_entities": extract_entities,
|
||||
"extract_relations": extract_relations,
|
||||
"extract_events": extract_events,
|
||||
"extract_triples": extract_triples,
|
||||
"resolve_coreferences": resolve_coreferences,
|
||||
}
|
||||
|
||||
return results
|
||||
|
||||
@@ -65,8 +65,6 @@ class NamedEntityRecognizer:
|
||||
• Handles multiple languages and domains
|
||||
• Processes batch text collections
|
||||
"""
|
||||
|
||||
<<<<<<< HEAD
|
||||
def __init__(
|
||||
self,
|
||||
methods: Optional[List[str]] = None,
|
||||
@@ -77,9 +75,6 @@ class NamedEntityRecognizer:
|
||||
config=None,
|
||||
**kwargs
|
||||
):
|
||||
=======
|
||||
def __init__(self, method=None, config=None, **kwargs):
|
||||
>>>>>>> origin/main
|
||||
"""
|
||||
Initialize named entity recognizer.
|
||||
|
||||
@@ -97,15 +92,12 @@ class NamedEntityRecognizer:
|
||||
self.config.update(kwargs)
|
||||
self.progress_tracker = get_progress_tracker()
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Store parameters
|
||||
self.methods = methods or ["spacy"]
|
||||
self.confidence_threshold = confidence_threshold
|
||||
self.merge_overlapping = merge_overlapping
|
||||
self.include_standard_types = include_standard_types
|
||||
|
||||
=======
|
||||
>>>>>>> origin/main
|
||||
# Use NERExtractor for actual extraction
|
||||
ner_config = self.config.get("ner", {})
|
||||
ner_config["confidence_threshold"] = confidence_threshold
|
||||
@@ -118,6 +110,7 @@ class NamedEntityRecognizer:
|
||||
self.entity_classifier = EntityClassifier(**self.config.get("classifier", {}))
|
||||
self.confidence_scorer = EntityConfidenceScorer(**self.config.get("scorer", {}))
|
||||
|
||||
|
||||
def extract_entities(self, text: str, **options) -> List[Entity]:
|
||||
"""
|
||||
Extract named entities from text.
|
||||
|
||||
@@ -87,8 +87,6 @@ class Relation:
|
||||
|
||||
class RelationExtractor:
|
||||
"""Relation extractor for entity relationships."""
|
||||
|
||||
<<<<<<< HEAD
|
||||
def __init__(
|
||||
self,
|
||||
method: Union[str, List[str]] = "pattern",
|
||||
@@ -98,9 +96,6 @@ class RelationExtractor:
|
||||
max_distance: int = 50,
|
||||
**config
|
||||
):
|
||||
=======
|
||||
def __init__(self, method: Union[str, List[str]] = "pattern", **config):
|
||||
>>>>>>> origin/main
|
||||
"""
|
||||
Initialize relation extractor.
|
||||
|
||||
@@ -129,15 +124,12 @@ class RelationExtractor:
|
||||
self.config = config
|
||||
self.progress_tracker = get_progress_tracker()
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Store parameters
|
||||
self.relation_types = relation_types
|
||||
self.bidirectional = bidirectional
|
||||
self.confidence_threshold = confidence_threshold
|
||||
self.max_distance = max_distance
|
||||
|
||||
=======
|
||||
>>>>>>> origin/main
|
||||
# Method configuration
|
||||
self.method = method if isinstance(method, list) else [method]
|
||||
self.min_confidence = config.get("min_confidence", confidence_threshold)
|
||||
@@ -163,6 +155,7 @@ class RelationExtractor:
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def extract_relations(
|
||||
self, text: str, entities: List[Entity], **options
|
||||
) -> List[Relation]:
|
||||
|
||||
@@ -19,33 +19,11 @@ This comprehensive guide demonstrates how to use the semantic extraction module
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Using the Convenience Function
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import build
|
||||
|
||||
text = "Apple Inc. was founded by Steve Jobs in 1976. The company is headquartered in Cupertino, California."
|
||||
|
||||
# Extract all semantic information
|
||||
result = build(
|
||||
text,
|
||||
extract_entities=True,
|
||||
extract_relations=True,
|
||||
extract_events=False,
|
||||
extract_triples=False,
|
||||
resolve_coreferences=False
|
||||
)
|
||||
|
||||
print(f"Extracted {len(result['entities'])} entities")
|
||||
print(f"Extracted {len(result['relations'])} relations")
|
||||
print(f"Statistics: {result['statistics']}")
|
||||
```
|
||||
|
||||
### Using Main Classes
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor
|
||||
|
||||
text = "Apple Inc. was founded by Steve Jobs in 1976. The company is headquartered in Cupertino, California."
|
||||
|
||||
# Extract entities
|
||||
ner = NamedEntityRecognizer()
|
||||
entities = ner.extract_entities(text)
|
||||
@@ -55,8 +33,11 @@ print(f"Entities: {entities}")
|
||||
rel_extractor = RelationExtractor()
|
||||
relations = rel_extractor.extract_relations(text, entities=entities)
|
||||
print(f"Relations: {relations}")
|
||||
|
||||
print(f"Extracted {len(entities)} entities and {len(relations)} relations")
|
||||
```
|
||||
|
||||
|
||||
## Entity Extraction
|
||||
|
||||
### Basic Entity Extraction
|
||||
@@ -677,23 +658,25 @@ print(f"Issues: {validation.issues}")
|
||||
### Building Knowledge Graph from Extraction
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import build
|
||||
from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripleExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
# Extract all information
|
||||
result = build(
|
||||
text,
|
||||
extract_entities=True,
|
||||
extract_relations=True,
|
||||
extract_triples=True
|
||||
)
|
||||
ner = NamedEntityRecognizer()
|
||||
entities = ner.extract_entities(text)
|
||||
|
||||
rel_extractor = RelationExtractor()
|
||||
relations = rel_extractor.extract_relations(text, entities=entities)
|
||||
|
||||
triple_extractor = TripleExtractor()
|
||||
triples = triple_extractor.extract_triples(text, entities=entities, relationships=relations)
|
||||
|
||||
# Build knowledge graph
|
||||
graph_builder = GraphBuilder()
|
||||
knowledge_graph = graph_builder.build({
|
||||
"entities": result["entities"],
|
||||
"relations": result["relations"],
|
||||
"triples": result["triples"]
|
||||
"entities": entities,
|
||||
"relations": relations,
|
||||
"triples": triples
|
||||
})
|
||||
|
||||
print(f"Knowledge graph nodes: {len(knowledge_graph.nodes)}")
|
||||
@@ -703,7 +686,7 @@ print(f"Knowledge graph edges: {len(knowledge_graph.edges)}")
|
||||
### Batch Processing
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import build
|
||||
from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor
|
||||
|
||||
texts = [
|
||||
"Apple Inc. was founded in 1976.",
|
||||
@@ -712,15 +695,17 @@ texts = [
|
||||
]
|
||||
|
||||
# Process multiple texts
|
||||
results = build(
|
||||
texts,
|
||||
extract_entities=True,
|
||||
extract_relations=True
|
||||
)
|
||||
ner = NamedEntityRecognizer()
|
||||
rel_extractor = RelationExtractor()
|
||||
|
||||
# Aggregate results
|
||||
all_entities = results["entities"]
|
||||
all_relations = results["relations"]
|
||||
all_entities = []
|
||||
all_relations = []
|
||||
|
||||
for text in texts:
|
||||
entities = ner.extract_entities(text)
|
||||
relations = rel_extractor.extract_relations(text, entities=entities)
|
||||
all_entities.extend(entities)
|
||||
all_relations.extend(relations)
|
||||
|
||||
print(f"Total entities: {len(all_entities)}")
|
||||
print(f"Total relations: {len(all_relations)}")
|
||||
|
||||
@@ -90,16 +90,12 @@ class TripleExtractor:
|
||||
"""RDF triple extraction handler."""
|
||||
|
||||
def __init__(
|
||||
<<<<<<< HEAD
|
||||
self,
|
||||
method: Union[str, List[str]] = "pattern",
|
||||
include_temporal: bool = False,
|
||||
include_provenance: bool = False,
|
||||
config=None,
|
||||
**kwargs
|
||||
=======
|
||||
self, method: Union[str, List[str]] = "pattern", config=None, **kwargs
|
||||
>>>>>>> origin/main
|
||||
):
|
||||
"""
|
||||
Initialize triple extractor.
|
||||
@@ -128,13 +124,10 @@ class TripleExtractor:
|
||||
self.config.update(kwargs)
|
||||
self.progress_tracker = get_progress_tracker()
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Store parameters
|
||||
self.include_temporal = include_temporal
|
||||
self.include_provenance = include_provenance
|
||||
|
||||
=======
|
||||
>>>>>>> origin/main
|
||||
# Method configuration
|
||||
self.method = method if isinstance(method, list) else [method]
|
||||
self.min_confidence = self.config.get("min_confidence", 0.5)
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
import json
|
||||
|
||||
# Read the notebook
|
||||
with open(r'c:\Users\Mohd Kaif\semantica\cookbook\use_cases\advanced_rag\01_GraphRAG_Complete.ipynb', 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Find and update the cell that uses build function (cell 14, lines 522-546)
|
||||
# Replace the build function usage with class-based approach
|
||||
data['cells'][14]['source'] = [
|
||||
"from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripleExtractor\n",
|
||||
"\n",
|
||||
"print(\"Extracting entities, relationships, and triples...\")\n",
|
||||
"\n",
|
||||
"ner = NamedEntityRecognizer()\n",
|
||||
"rel_extractor = RelationExtractor()\n",
|
||||
"triple_extractor = TripleExtractor()\n",
|
||||
"\n",
|
||||
"flat_entities = []\n",
|
||||
"flat_relationships = []\n",
|
||||
"flat_triples = []\n",
|
||||
"\n",
|
||||
"for doc in normalized_documents:\n",
|
||||
" text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n",
|
||||
" \n",
|
||||
" entities = ner.extract_entities(text)\n",
|
||||
" flat_entities.extend(entities if isinstance(entities, list) else [entities])\n",
|
||||
" \n",
|
||||
" relations = rel_extractor.extract_relations(text, entities=entities)\n",
|
||||
" flat_relationships.extend(relations if isinstance(relations, list) else [relations])\n",
|
||||
" \n",
|
||||
" triples = triple_extractor.extract_triples(text, entities=entities, relationships=relations)\n",
|
||||
" flat_triples.extend(triples if isinstance(triples, list) else [triples])\n",
|
||||
"\n",
|
||||
"print(f\"Extracted {len(flat_entities)} entities\")\n",
|
||||
"print(f\"Extracted {len(flat_relationships)} relationships\")\n",
|
||||
"print(f\"Extracted {len(flat_triples)} triples\")\n"
|
||||
]
|
||||
|
||||
# Write back the notebook
|
||||
with open(r'c:\Users\Mohd Kaif\semantica\cookbook\use_cases\advanced_rag\01_GraphRAG_Complete.ipynb', 'w', encoding='utf-8') as f:
|
||||
json.dump(data, f, indent=1, ensure_ascii=False)
|
||||
|
||||
print("Updated notebook successfully!")
|
||||
Reference in New Issue
Block a user