semantica/docs/arrow_exporter.md

Apache Arrow Exporter

Overview

The Apache Arrow exporter provides high-performance columnar data export for Semantica's knowledge graphs, entities, and relationships. It uses explicit schemas (no inference) and writes Arrow IPC files (.arrow) that are compatible with Pandas and DuckDB.

Features

• Explicit Schemas: Pre-defined schemas for entities and relationships (no inference)
• Columnar Format: Efficient storage and fast analytics
• Metadata Support: Converts metadata dictionaries to Arrow struct fields
• Field Normalization: Handles various entity and relationship field name variations
• Progress Tracking: Integrated progress monitoring
• Error Handling: Structured error handling with detailed logging
• Pandas/DuckDB Compatible: Direct conversion to DataFrames and SQL queries

Installation

The Arrow exporter requires PyArrow:

pip install pyarrow

Usage

Basic Usage

from semantica.export import ArrowExporter

# Initialize exporter
exporter = ArrowExporter()

# Export entities
entities = [
    {"id": "e1", "text": "Alice", "type": "Person", "confidence": 0.95},
    {"id": "e2", "text": "Acme Corp", "type": "Organization", "confidence": 0.88}
]
exporter.export_entities(entities, "entities.arrow")

# Export relationships
relationships = [
    {"id": "r1", "source_id": "e1", "target_id": "e2", "type": "WORKS_FOR"}
]
exporter.export_relationships(relationships, "relationships.arrow")

# Export knowledge graph
knowledge_graph = {
    "entities": entities,
    "relationships": relationships
}
exporter.export_knowledge_graph(knowledge_graph, "kg_base")
# Creates: kg_base_entities.arrow, kg_base_relationships.arrow

Using Convenience Function

from semantica.export import export_arrow

# Simple export
export_arrow(entities, "entities.arrow")

# Export multiple types
data = {
    "entities": entities,
    "relationships": relationships
}
export_arrow(data, "output_base")

With Compression

# Use LZ4 compression
exporter = ArrowExporter(compression="lz4")
exporter.export_entities(entities, "entities_compressed.arrow")

Schemas

Entity Schema

ENTITY_SCHEMA = pa.schema([
    pa.field("id", pa.string(), nullable=False),
    pa.field("text", pa.string(), nullable=True),
    pa.field("type", pa.string(), nullable=True),
    pa.field("confidence", pa.float64(), nullable=True),
    pa.field("start", pa.int64(), nullable=True),
    pa.field("end", pa.int64(), nullable=True),
    pa.field("metadata", pa.struct([
        pa.field("keys", pa.list_(pa.string())),
        pa.field("values", pa.list_(pa.string()))
    ]), nullable=True),
])

Relationship Schema

RELATIONSHIP_SCHEMA = pa.schema([
    pa.field("id", pa.string(), nullable=False),
    pa.field("source_id", pa.string(), nullable=False),
    pa.field("target_id", pa.string(), nullable=False),
    pa.field("type", pa.string(), nullable=True),
    pa.field("confidence", pa.float64(), nullable=True),
    pa.field("metadata", pa.struct([
        pa.field("keys", pa.list_(pa.string())),
        pa.field("values", pa.list_(pa.string()))
    ]), nullable=True),
])

Field Normalization

The exporter automatically normalizes field names:

Entities:

• text, label, name → text
• type, entity_type → type
• id, entity_id → id
• start, start_offset → start
• end, end_offset → end

Relationships:

• source, source_id → source_id
• target, target_id → target_id
• type, relationship_type → type

Reading Arrow Files

With PyArrow

import pyarrow as pa
import pyarrow.ipc as ipc

with pa.OSFile("entities.arrow", 'rb') as source:
    with ipc.open_file(source) as reader:
        table = reader.read_all()
        print(table.schema)
        print(table.to_pandas())

With Pandas

import pandas as pd
import pyarrow.ipc as ipc

with ipc.open_file("entities.arrow") as reader:
    df = reader.read_all().to_pandas()
    print(df)

With DuckDB

import duckdb

# Query Arrow file directly
result = duckdb.query("SELECT * FROM 'entities.arrow' WHERE type = 'Person'")
print(result.df())

Methods

export(data, file_path, schema=None, **options)

Generic export method that handles both single and multiple files.

Parameters:

• data: List of dicts or dict with list values
• file_path: Output file path (base path for dict exports)
• schema: Optional Arrow schema (auto-detected if not provided)
• **options: Additional options

export_entities(entities, file_path, **options)

Export entities to Arrow IPC file with normalization.

Parameters:

• entities: List of entity dictionaries
• file_path: Output Arrow file path
• **options: Additional options

export_relationships(relationships, file_path, **options)

Export relationships to Arrow IPC file with normalization.

Parameters:

• relationships: List of relationship dictionaries
• file_path: Output Arrow file path
• **options: Additional options

export_knowledge_graph(knowledge_graph, base_path, **options)

Export knowledge graph to multiple Arrow files.

Parameters:

• knowledge_graph: Knowledge graph dictionary with 'entities' and 'relationships'
• base_path: Base path for output files (without extension)
• **options: Additional options

Examples

See examples/arrow_export_example.py for comprehensive usage examples.

Testing

Run the test suite:

# All Arrow exporter tests
pytest tests/test_arrow_exporter.py -v

# Integration tests
pytest tests/test_export_module.py::TestExportModule::test_arrow_exporter -v

Performance Benefits

• Columnar Storage: Faster analytics on specific columns
• Compression: Smaller file sizes (especially with LZ4/ZSTD)
• Zero-Copy: Memory-efficient data transfer
• Cross-Language: Works with Python, R, Julia, JavaScript, and more
• SQL Queries: Direct querying with DuckDB without loading into memory

Comparison with Other Formats

Feature	Arrow	CSV	JSON
Type Safety	✓	✗	✗
Compression	✓	✗	✗
Schema Validation	✓	✗	✗
Pandas Compatible	✓	✓	✓
DuckDB Native	✓	✓	✗
Binary Format	✓	✗	✗
Human Readable	✗	✓	✓

Architecture

The Arrow exporter follows Semantica's export architecture:

1. Normalization: Field names are normalized to consistent format
2. Schema Application: Explicit schemas ensure type safety
3. Metadata Conversion: Dicts converted to Arrow struct fields
4. Progress Tracking: Integrated with Semantica's progress tracker
5. Error Handling: Structured exceptions with detailed messages

Contributing

When contributing to the Arrow exporter:

1. Maintain explicit schemas (no inference)
2. Follow existing code style and patterns
3. Add comprehensive tests for new features
4. Update this documentation
5. Ensure Pandas/DuckDB compatibility

License

MIT License - See LICENSE file for details.

Author

Semantica Contributors