semantica/docs/reference/seed.md

title, description, icon

title	description	icon
Seed Module	Bootstrap Knowledge Graphs from verified, structured sources — taxonomies, reference tables, product catalogs, and domain anchors.	database

semantica.seed gives your knowledge graph a reliable starting point. Rather than building from an empty graph and hoping extraction produces consistent reference data, you load verified, structured sources first — ISO codes, employee rosters, product catalogs, domain taxonomies — then merge freshly extracted data on top.

Exported Classes

Class	Role
SeedDataManager	Coordinator: register_source, load_all, create_foundation_graph, integrate_with_extracted
SeedDataSource	Config dataclass: {name, source_type, path, config} — one per registered source
SeedData	Loaded data container: {entities, relationships, metadata} returned by load_all

What You Get

Register sources, build a foundation graph, validate quality, and merge with extracted data. Typed source definition supporting CSV, JSON, SQL, API, and RDF with format-specific config. Build a foundation graph from all registered sources in one pass, ready to merge with extracted data. `seed_first`, `extracted_first`, and `smart_merge` with property-level conflict detection. Required field checks, ID uniqueness, type consistency, reference integrity, and encoding validation before loading. Track seed data versions across pipeline runs and diff changes between versions. **When to use the Seed Module:** Bootstrapping with structured reference data (taxonomies, user lists, product catalogs), loading immutable facts (ISO country codes, standard ontology terms) that extracted data should not override, ensuring test reproducibility with deterministic datasets, and anchoring entity disambiguation with canonical forms.

Quick Start

```python from semantica.seed import SeedDataManager

manager = SeedDataManager()

manager.register_source("countries",  "csv",  "data/countries.csv")
manager.register_source("taxonomy",   "json", "data/taxonomy.json")
manager.register_source("employees",  "csv",  "data/employees.csv")
```

```python foundation_kg = manager.create_foundation_graph()

print(f"Foundation nodes: {foundation_kg.node_count}")
print(f"Foundation edges: {foundation_kg.edge_count}")
```

```python report = manager.validate_quality(manager.load_source("employees"))

if not report.is_valid:
    for issue in report.issues:
        print(f"[{issue.severity}] Row {issue.row}: {issue.message}")
else:
    print(f"Validated {report.record_count} records — no issues found")
```

```python final_kg = manager.integrate_with_extracted( seed_graph=foundation_kg, extracted_data=new_entities, strategy="smart_merge", ) print(f"Final graph: {final_kg.node_count} nodes, {final_kg.edge_count} edges") ```

SeedDataSource Types

```python from semantica.seed import SeedDataSource

csv_source = SeedDataSource(
    name="employees",
    type="csv",
    path="data/employees.csv",
    config={
        "delimiter": ";",
        "encoding":  "utf-8",
        "id_column": "employee_id",
        "type":      "Person",
    }
)

manager.register_source("employees", "csv", csv_source.path, config=csv_source.config)
```

Best for: employee rosters, product lists, reference tables.

```python json_source = SeedDataSource( name="taxonomy", type="json", path="data/taxonomy.json", config={"encoding": "utf-8"} ) ```

Expects an array of entity objects. Best for: taxonomies, ontology term lists, structured configs.

```python sql_source = SeedDataSource( name="products", type="sql", path="postgresql://user:pass@localhost/db", config={"query": "SELECT id, name, category FROM products WHERE active = true"} ) ```

Best for: live database tables — PostgreSQL, MySQL, SQLite.

```python # API source — fetch from a REST endpoint api_source = SeedDataSource( name="geo_codes", type="api", path="https://restcountries.com/v3.1/all", config={"fields": ["name", "cca2", "region"]} )

# RDF source — OWL ontologies or Turtle files
rdf_source = SeedDataSource(
    name="domain_ontology",
    type="rdf",
    path="data/ontology.ttl",
    config={"format": "turtle"}
)
```

API: external reference APIs (countries, currencies, geo). RDF: existing knowledge bases, OWL ontologies.

| Type | Path Format | Use Case |
| ---- | ----------- | -------- |
| `csv` | File path | Employee rosters, product lists, reference tables |
| `json` | File path | Taxonomies, ontology term lists, structured configs |
| `sql` | Connection string | Live database tables — PostgreSQL, MySQL, SQLite |
| `api` | URL | External reference APIs (countries, currencies, geo) |
| `rdf` | File path | OWL ontologies, Turtle files, existing knowledge bases |

SeedDataManager Reference

Method	Description
register_source(name, format, path)	Add a named data source to the registry
create_foundation_graph()	Build a KG from all registered sources
validate_quality(seed_data)	Check schema compliance, required fields, and duplicates
integrate_with_extracted(seed, extracted, strategy)	Merge seed and extracted graphs
export_seed_data(path, format)	Export seed graph to RDF (turtle, json-ld), JSON, or CSV
load_from_csv(path)	Load seed records from a CSV file
load_from_json(path)	Load seed records from a JSON file
list_sources()	List all registered source names and their formats
get_version(name)	Get the current version metadata for a named source

Merge Strategies

Seed data wins on every conflicting property. Use when seed encodes authoritative reference facts that must not be overridden.

```python
final_kg = manager.integrate_with_extracted(
    seed_graph=foundation_kg,
    extracted_data=new_entities,
    strategy="seed_first",
)
```

Best for: ISO codes, canonical entity names, official taxonomy IDs, employee records.

Extracted data overrides seed on conflicting properties. Use when new documents contain more current information than your reference data.

```python
final_kg = manager.integrate_with_extracted(
    seed_graph=foundation_kg,
    extracted_data=new_entities,
    strategy="extracted_first",
)
```

Best for: frequently changing attributes like addresses, titles, revenue figures.

Property-level merge with conflict detection — irresolvable conflicts are logged for manual review rather than silently overwritten.

```python
final_kg = manager.integrate_with_extracted(
    seed_graph=foundation_kg,
    extracted_data=new_entities,
    strategy="smart_merge",
)
```

Best for: general-purpose pipelines where surfacing conflicts is more valuable than silently losing data.

Built-in Datasets

Register built-in reference datasets as named sources and load them into your foundation graph:

from semantica.seed import SeedDataManager

manager = SeedDataManager()

# Register built-in reference sources by format and path
manager.register_source("countries",  "csv",  "data/iso_countries.csv")
manager.register_source("currencies", "json", "data/iso_currencies.json")

foundation_kg = manager.create_foundation_graph()

Dataset	Content
companies	Fortune 500 companies with type, sector, HQ
countries	ISO 3166 country codes, regions, populations
currencies	ISO 4217 codes, symbols, names
person_names	Common first/last names for synthetic data

Full Pipeline Example

from semantica.seed import SeedDataManager
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.split import TextSplitter
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
from semantica.llms import Groq
import os

llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))

# Step 1 — Build the foundation from verified reference data
seed_manager = SeedDataManager()
seed_manager.register_source("taxonomy",  "json", "data/taxonomy.json")
seed_manager.register_source("employees", "csv",  "data/employees.csv")
foundation_kg = seed_manager.create_foundation_graph()

# Step 2 — Ingest and extract from unstructured documents
ingestor  = FileIngestor()
parser    = DocumentParser()
splitter  = TextSplitter(method="semantic_transformer", chunk_size=512)
ner       = NERExtractor(method="llm", llm_provider=llm)
rel_ext   = RelationExtractor(method="llm", llm_provider=llm)

sources                 = ingestor.ingest("news_articles/")
extracted_entities      = []
extracted_relationships = []

for source in sources:
    parsed = parser.parse(source)
    chunks = splitter.split_document(parsed)
    for chunk in chunks:
        entities      = ner.extract(chunk.text)
        relationships = rel_ext.extract(chunk.text, entities=entities)
        extracted_entities.extend(entities)
        extracted_relationships.extend(relationships)

# Step 3 — Merge seed and extracted data
final_kg = seed_manager.integrate_with_extracted(
    seed_graph=foundation_kg,
    extracted_data=extracted_entities,
    strategy="seed_first",
)
print(f"Final graph: {final_kg.node_count} nodes, {final_kg.edge_count} edges")

Versioning

Track seed data versions to detect when reference data changes between pipeline runs:

manager = SeedDataManager()
manager.register_source("taxonomy", "json", "data/taxonomy.json")

version = manager.get_version("taxonomy")
print(f"Version: {version.version_id}")
print(f"Hash:    {version.checksum}")
print(f"Records: {version.record_count}")
print(f"Updated: {version.last_modified}")

YAML Configuration

Define sources in YAML for production deployments — no code changes needed to switch environments:

seed:
  sources:
    - name: "employees"
      type: "csv"
      path: "./data/employees.csv"
      config:
        id_column: "employee_id"
        type: "Person"
    - name: "taxonomy"
      type: "json"
      path: "./data/taxonomy.json"
    - name: "products"
      type: "sql"
      path: "${DATABASE_URL}"
      config:
        query: "SELECT id, name, category FROM products WHERE active = true"
  merge:
    strategy: "smart_merge"
  validation:
    strict: true
    required_fields: ["id", "type"]

Environment variable overrides:

export SEMANTICA_SEED_DATA_DIR=./data/seed
export SEMANTICA_SEED_MERGE_STRATEGY=seed_first

Tips and Common Pitfalls

**Load seed data before extracted data.** Seed data is your ground truth — normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values. **Use `seed_first` merge strategy for reference data.** When seed data encodes authoritative facts (official company names, canonical taxonomy IDs, employee records), `strategy="seed_first"` ensures those values win over extracted values. Use `smart_merge` only when extracted data may be more current than the seed. **Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast — always run it first. **Register all sources before calling `create_foundation_graph()`.** `create_foundation_graph()` processes all registered sources in one pass. Registering a source after calling it means that source is silently excluded. Register all sources at the start of your script, then call `create_foundation_graph()` once. **Track seed versions to detect drift.** Use `manager.get_version()` to check the checksum and record count for each registered source between pipeline runs. If a taxonomy file changes, downstream entity normalisation and deduplication thresholds may need re-tuning — don't treat seed data as static. **Use YAML configuration for production deployments.** Hard-coding source paths in Python scripts makes environment-switching (dev → staging → prod) fragile. Declare sources in `config.yaml` under the `seed:` key and override paths with `SEMANTICA_SEED_DATA_DIR`. This way, the same code runs in every environment. Load unstructured data alongside seed data. The target graph that seed data populates. Handle duplicates during seed-extracted merge. Incorporate seed loading as a named pipeline step.