semantica/docs/reference/semantic_extract.md

# Semantic Extract

> **Advanced information extraction system for Entities, Relations, Events, and Triples.**

---

## 🎯 Overview

<div class="grid cards" markdown>

-   :material-account-search:{ .lg .middle } **NER**

    ---

    Extract Named Entities (Person, Org, Loc) with confidence scores

-   :material-relation-one-to-one:{ .lg .middle } **Relation Extraction**

    ---

    Identify relationships between entities (e.g., `founded_by`, `located_in`)

-   :material-calendar-clock:{ .lg .middle } **Event Detection**

    ---

    Detect events with temporal information and participants

-   :material-format-quote-close:{ .lg .middle } **Coreference**

    ---

    Resolve pronouns ("he", "it") to their entity references

-   :material-share-variant:{ .lg .middle } **Triple Extraction**

    ---

    Extract Subject-Predicate-Object triples for Knowledge Graphs

-   :material-robot:{ .lg .middle } **LLM Enhancement**

    ---

    Use LLMs to improve extraction quality and handle complex schemas

</div>

!!! tip "When to Use"
    - **KG Construction**: Converting unstructured text into structured graph data
    - **Text Analysis**: Identifying key actors and events in documents
    - **Search Indexing**: Extracting metadata for faceted search
    - **Data Enrichment**: Adding semantic tags to content

---

## ⚙️ Algorithms Used

### Named Entity Recognition (NER)
- **Transformer Models**: BERT/RoBERTa for token classification.
- **Regex Patterns**: Pattern matching for specific formats (Emails, IDs).
- **LLM Prompting**: Zero-shot extraction for custom entity types.

### Relation Extraction
- **Dependency Parsing**: Analyzing grammatical structure to find subject-verb-object paths.
- **Joint Extraction**: Extracting entities and relations simultaneously.
- **Semantic Role Labeling**: Identifying "Who did What to Whom".

### Coreference Resolution
- **Mention Detection**: Finding all potential references (nouns, pronouns).
- **Clustering**: Grouping mentions that refer to the same real-world entity.
- **Pronoun Resolution**: Mapping pronouns to the most likely antecedent.

### Triple Extraction
- **OpenIE**: Open Information Extraction for arbitrary relation strings.
- **Schema-Based**: Mapping extracted relations to a predefined ontology.
- **Reification**: Handling complex relations (time, location) by creating event nodes.

---

## Main Classes

### NamedEntityRecognizer

Coordinator for entity extraction.

**Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `methods` | list | `["spacy"]` | Extraction methods to use |
| `confidence_threshold` | float | `0.5` | Minimum confidence score |
| `merge_overlapping` | bool | `True` | Merge overlapping entities |
| `include_standard_types` | bool | `True` | Include Person, Org, Location |

**Methods:**

| Method | Description |
|--------|-------------|
| `extract_entities(text)` | Get list of entities |
| `add_custom_pattern(pattern)` | Add regex rule |

**Example:**

```python
from semantica.semantic_extract import NamedEntityRecognizer

# Basic usage
ner = NamedEntityRecognizer()
entities = ner.extract_entities("Elon Musk leads SpaceX.")
# [Entity(text="Elon Musk", label="PERSON"), Entity(text="SpaceX", label="ORG")]

# With configuration
ner = NamedEntityRecognizer(
    methods=["spacy", "rule-based"],
    confidence_threshold=0.7,
    merge_overlapping=True
)
entities = ner.extract_entities("Apple Inc. was founded in 1976.")
```

### RelationExtractor

Extracts relationships between entities.

**Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `relation_types` | list | `None` | Specific relation types to extract |
| `bidirectional` | bool | `False` | Extract bidirectional relations |
| `confidence_threshold` | float | `0.6` | Minimum confidence score |
| `max_distance` | int | `50` | Max token distance between entities |

**Methods:**

| Method | Description |
|--------|-------------|
| `extract_relations(text, entities)` | Find links |

**Example:**

```python
from semantica.semantic_extract import RelationExtractor, NamedEntityRecognizer

# First extract entities
ner = NamedEntityRecognizer()
text = "Elon Musk founded SpaceX in 2002."
entities = ner.extract_entities(text)

# Basic relation extraction
rel_extractor = RelationExtractor()
relations = rel_extractor.extract_relations(text, entities=entities)
# [Relation(source="Elon Musk", target="SpaceX", type="founded")]

# With configuration
rel_extractor = RelationExtractor(
    relation_types=["founded", "leads", "works_at"],
    confidence_threshold=0.7,
    bidirectional=False
)
relations = rel_extractor.extract_relations(text, entities=entities)
```

### EventDetector

Identifies events with temporal information and participants.

**Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `event_types` | list | `None` | Specific event types to detect |
| `extract_participants` | bool | `True` | Extract event participants |
| `extract_location` | bool | `True` | Extract event locations |
| `extract_time` | bool | `True` | Extract temporal information |

**Methods:**

| Method | Description |
|--------|-------------|
| `detect_events(text)` | Find events |

**Example:**

```python
from semantica.semantic_extract import EventDetector

detector = EventDetector(
    event_types=["launch", "acquisition", "announcement"],
    extract_participants=True,
    extract_time=True
)
events = detector.detect_events("SpaceX launched Starship on March 14, 2024.")
```

### TripleExtractor

Extracts RDF triples (Subject-Predicate-Object).

**Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `include_temporal` | bool | `False` | Include time information |
| `include_provenance` | bool | `False` | Track source sentences |

**Methods:**

| Method | Description |
|--------|-------------|
| `extract_triples(text)` | Get (S, P, O) tuples |

**Example:**

```python
from semantica.semantic_extract import TripleExtractor

extractor = TripleExtractor(
    include_temporal=True,
    include_provenance=True
)
triples = extractor.extract_triples("Steve Jobs founded Apple in 1976.")
# [Triple(subject="Steve Jobs", predicate="founded", object="Apple", temporal="1976")]
```

---

## Usage Examples

```python
from semantica.semantic_extract import (
    NamedEntityRecognizer, 
    RelationExtractor,
    TripleExtractor,
    EventDetector,
    CoreferenceResolver
)

text = "Apple released the iPhone in 2007. Steve Jobs announced it at Macworld."

# Extract entities with confidence filtering
ner = NamedEntityRecognizer(confidence_threshold=0.7)
entities = ner.extract_entities(text)

# Resolve coreferences (recommended before relation extraction)
coref = CoreferenceResolver()
resolved = coref.resolve(text)

# Extract relations
rel_extractor = RelationExtractor(confidence_threshold=0.6)
relations = rel_extractor.extract_relations(text, entities=entities)

# Extract triples for KG
triple_extractor = TripleExtractor(include_temporal=True)
triples = triple_extractor.extract_triples(text)

# Detect events
event_detector = EventDetector(extract_time=True)
events = event_detector.detect_events(text)

print(f"Entities: {len(entities)}")
print(f"Relations: {len(relations)}")
print(f"Triples: {len(triples)}")
print(f"Events: {len(events)}")
```

---

## Configuration

### Environment Variables

```bash
export NER_MODEL=dslim/bert-base-NER
export RELATION_MODEL=semantica/rel-extract-v1
export EXTRACT_CONFIDENCE_THRESHOLD=0.7
```

### YAML Configuration

```yaml
semantic_extract:
  ner:
    model: dslim/bert-base-NER
    min_confidence: 0.7
    
  relations:
    max_distance: 50 # tokens
    
  coreference:
    enabled: true
```

---

## Integration Examples

### KG Population Pipeline

```python
from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripleExtractor
from semantica.kg import GraphBuilder

# 1. Extract
text = "Google was founded by Larry Page and Sergey Brin."
ner = NamedEntityRecognizer()
entities = ner.extract_entities(text)
triple_extractor = TripleExtractor()
triples = triple_extractor.extract_triples(text)

# 2. Populate KG using GraphBuilder
builder = GraphBuilder()
sources = [{
    "entities": entities,
    "relationships": [{"source": t.subject, "target": t.object, "type": t.predicate} for t in triples]
}]
kg = builder.build(sources)
```

---

## Best Practices

1.  **Resolve Coreferences**: Always run coreference resolution *before* relation extraction to link "He" to "John Doe".
2.  **Filter Low Confidence**: Set a confidence threshold (e.g., 0.7) to reduce noise.
3.  **Use Custom Patterns**: For domain-specific IDs (e.g., "Invoice #123"), regex is faster and more accurate than ML.
4.  **Batch Processing**: Use batch methods when processing large corpora.

---

## See Also

- [Parse Module](parse.md) - Prepares text for extraction
- [Ontology Module](ontology.md) - Defines the schema for extraction
- [Knowledge Graph Module](kg.md) - Stores the extracted data