semantica/docs/concepts.md

title, description, icon

title	description	icon
Core Concepts	The fundamental ideas behind Semantica — knowledge graphs, reasoning, provenance, and temporal intelligence explained.	book-open

New here? Start with [Getting Started](getting-started) for hands-on examples, then come back here for deeper understanding.

---

What is Semantica?

Semantica transforms unstructured data (documents, web pages, reports, databases) into knowledge graphs — structured representations that AI systems can query, reason about, and trace back to sources.

At its core, Semantica adds a context and intelligence layer on top of your existing AI stack: it doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs accountable.

---

Knowledge Graphs

The foundation of everything in Semantica.

A knowledge graph stores information as:

• Nodes (entities) — people, companies, locations, events, concepts
• Edges (relationships) — works_for, located_in, founded_by
• Properties — name, date, confidence score, source URL

This structure makes knowledge searchable, connectable, queryable, and — critically — explainable: every answer can be traced back to the facts and relationships that produced it.

---

Entity Extraction (NER)

Scanning text to find and classify real-world entities.

# Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
{
    "entities": [
        {"text": "Apple Inc.",  "type": "ORGANIZATION", "confidence": 0.98},
        {"text": "Steve Jobs",  "type": "PERSON",       "confidence": 0.99},
        {"text": "1976",        "type": "DATE",         "confidence": 0.95},
        {"text": "Cupertino",   "type": "LOCATION",     "confidence": 0.97}
    ]
}

Each entity gets a type, confidence score, and a link to its source document.

---

Relationship Extraction

Finding how entities connect to each other.

{
    "relationships": [
        {"subject": "Steve Jobs", "predicate": "founded",    "object": "Apple Inc.", "confidence": 0.92},
        {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino",  "confidence": 0.89}
    ]
}

Relationships can be extracted via rule-based methods, ML models, or LLMs (with "llm_typed" metadata).

---

Embeddings

Embeddings convert text into numerical vectors so that AI systems can measure semantic similarity — finding related concepts even when the exact words differ.

Semantica uses embeddings for:

• Semantic search — retrieve by meaning, not just keywords
• Entity resolution — match the same entity across different sources
• Precedent search — find similar past decisions
• GraphRAG retrieval — hybrid vector + graph traversal

---

GraphRAG

GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses by grounding them in a structured knowledge graph rather than raw text chunks alone.

How it works:

1. User submits a query
2. Semantica retrieves relevant graph context (entities, relationships, reasoning paths)
3. The LLM generates a response grounded in that context
4. Every claim in the response links back to a source node in the graph

This eliminates the hallucination and traceability problems of standard RAG.

---

Ontology

An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply.

ontology = {
    "classes": ["Person", "Organization", "Location"],
    "relationships": ["works_for", "located_in", "founded_by"],
    "rules": {
        "Person":       ["must_have_name"],
        "Organization": ["must_have_name", "can_have_founding_date"]
    }
}

Semantica can auto-generate ontologies from your knowledge graph, or import existing OWL/RDF/Turtle ontologies. The Ontology Hub (v0.5.0) provides a visual editor, SHACL Studio, alignment authoring, and a health dashboard.

---

Reasoning & Inference

Semantica includes multiple reasoning engines to derive new knowledge from existing facts.

Known:    Steve Jobs founded Apple Inc.
Known:    Apple Inc. is headquartered in Cupertino
Inferred: Steve Jobs has a connection to Cupertino

Supported engines:

Engine	Description
Forward chaining	Applies rules repeatedly until no new facts can be derived
Rete network	Efficient pattern matching for large rule sets
Deductive	Classical deductive reasoning
Abductive	Infers the most likely explanation
SPARQL	Query-based inference over RDF graphs
Datalog	Recursive Horn clause rules with fixpoint semantics (v0.4.0)

All engines produce explainable inference paths, not black-box conclusions.

---

Temporal Intelligence (v0.4.0)

Knowledge changes over time. Temporal graphs attach valid_from / valid_until windows to nodes and edges, enabling point-in-time queries and historical analysis.

from semantica.kg import TemporalKnowledgeGraph
from datetime import datetime

tkg = TemporalKnowledgeGraph()
tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1))

# Query the graph as it existed on a specific date
snapshot = tkg.at(datetime(2021, 6, 15))

Features: Allen interval algebra (all 13 relations), OWL-Time export, recorded_at stamping, temporal provenance.

Common uses: tracking company leadership changes, policy evolution, research timelines, financial instrument histories.

---

Distance Intelligence (v0.5.0)

Explore the semantic neighborhood of any entity in your graph.

from semantica.kg import DistanceCalculator

calc = DistanceCalculator(graph)
neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4)
matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"])

Features: N×N distance matrices, ego-mode visualization, distance band classification (near / mid / far), embedding cache optimization.

---

Deduplication & Entity Resolution

Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance.

Strategies: Jaro-Winkler similarity (v1), blocking_v2, hybrid_v2, semantic_v2 (v2 — up to 7x faster).

---

Provenance & Auditability

Every fact in Semantica links back to:

• The source document it came from
• The extraction method used
• The ontology rules applied
• The reasoning steps that produced any inference

This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11).

---

Decision Intelligence

Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent.

decision_id = context.record_decision(
    category="model_selection",
    scenario="Choose LLM for production pipeline",
    reasoning="GPT-4 benchmark advantage justifies 3x cost increase",
    outcome="selected_gpt4",
    confidence=0.91,
)

precedents = context.find_precedents("model selection reasoning", limit=5)
influence  = context.analyze_decision_influence(decision_id)

---

Conflict Detection

When multiple sources disagree on the same fact, Semantica flags and resolves the conflict rather than silently picking one value.

Resolution strategies: prefer most recent, prefer most reliable source, majority vote, or flag for manual review.

---

Next Steps

Build a full pipeline with code. Every module explained. Real-world domain examples. Complete technical reference.