diff --git a/docs/architecture.md b/docs/architecture.md index d2ab458d..80531aa3 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -8,18 +8,7 @@ Semantica is built around a three-layer modular architecture. Import only what y ## Three-Layer Architecture -```text -┌─────────────────────────────────────────────────────────────────┐ -│ Layer 1: Data Ingestion │ -│ Files · Web · APIs · Databases · Streams │ -├─────────────────────────────────────────────────────────────────┤ -│ Layer 2: Semantic Processing │ -│ Parse · Normalize · Extract · Build · QA │ -├─────────────────────────────────────────────────────────────────┤ -│ Layer 3: Application │ -│ GraphRAG · AI Agents · Analytics · Export · Visualization │ -└─────────────────────────────────────────────────────────────────┘ -``` +Semantica four-layer architecture @@ -80,16 +69,7 @@ Consumes the knowledge graph for downstream use cases. Every pipeline follows the same linear path from raw source to delivered output: -```text -Ingest → raw data from files, web, databases, streams -Parse → structured text and layout extraction -Normalize → canonical forms, date and name standardization -Extract → entities, relationships, events, triplets -Build → entity resolution and graph construction -QA → deduplication, conflict resolution, validation -Store → vector store, graph store, triplet store -Deliver → GraphRAG, agents, export, visualization -``` +Semantica 8-step pipeline: Ingest → Parse → Normalize → Extract → Build KG → QA → Store → Deliver ## Module Map diff --git a/docs/assets/img/diagrams/agent-context-flow.svg b/docs/assets/img/diagrams/agent-context-flow.svg new file mode 100644 index 00000000..789c05fa --- /dev/null +++ b/docs/assets/img/diagrams/agent-context-flow.svg @@ -0,0 +1,109 @@ + +AgentContext Memory and Decision Flow + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +AgentContext — Memory, Decisions & Graph Intelligence + + + + +AGENT +AI Agent +store(fact) +retrieve(query) +record_decision() + + + + +calls +results + + + + +CORE MODULE +AgentContext + + + +store(content) + + +retrieve(query) + + +record_decision() + + +find_precedents() + + +analyze_influence() + + +get_causal_chain() + + + +embed + index + + + +top-k results + + + +store decision + + + +causal chain + + + + +VectorStore +Semantic memory +FAISS · Pinecone · Qdrant +embedding-backed retrieval + + + + +ContextGraph +Decision graph +causal chains · precedents +centrality · community + + +decision_tracking=True enables full audit trail — every agent action is recorded and traceable + diff --git a/docs/assets/img/diagrams/architecture-overview.svg b/docs/assets/img/diagrams/architecture-overview.svg new file mode 100644 index 00000000..56077fce --- /dev/null +++ b/docs/assets/img/diagrams/architecture-overview.svg @@ -0,0 +1,159 @@ + +Semantica Architecture Overview + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +INGESTION +Data Sources + + +Files & PDFs +FileIngestor · DocxParser + + + +Web & APIs +WebIngestor · REST + + + +Databases +DBIngestor · DuckDB + + + +Streams +StreamIngestor · Kafka + + + +Parquet & XML +PyArrow · lxml · v0.5.0 + + + + + + + +PROCESSING +Semantic Engine + + +Parse & Layout +DocumentParser · Docling + + + +Normalize +TextNormalizer · dates + + + +NER & Relations +pattern · ml · llm + + + +Build Graph +GraphBuilder · merge + + + +QA & Dedup +ConflictDetector · merge + + + + + + + +INTELLIGENCE +Knowledge Store + + +Knowledge Graph +NetworkX · Neo4j · AGE + + + +Vector Store +FAISS · Pinecone · Qdrant + + + +Ontology +OWL · SHACL · SKOS + + + +Triplet Store +SPARQL · Blazegraph · Jena + + + +Embeddings +ST · FastEmbed · BGE + + + + + + + +APPLICATION +Output Layer + + +GraphRAG Agents +AgentContext · memory + + + +Decision Tracking +causal chains · audit + + + +Reasoning +forward chain · Datalog + + + +Explorer & Hub +SHACL Studio · Ontology + + + +Export & Provenance +RDF · Parquet · PROV-O + diff --git a/docs/assets/img/diagrams/extraction-pipeline.svg b/docs/assets/img/diagrams/extraction-pipeline.svg new file mode 100644 index 00000000..e6c36d0f --- /dev/null +++ b/docs/assets/img/diagrams/extraction-pipeline.svg @@ -0,0 +1,82 @@ + +Semantic Extraction Pipeline + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +All extractors support pattern · ml · llm modes — swap with one parameter change + + + + +INPUT +Raw Text +"Apple Inc. was founded by Steve Jobs in 1976" + + + + + + + + + +NERExtractor +Entities +PERSON · ORG · LOC · DATE + + + + +RelationExtractor +Relations +founded · located_in · works_for + + + + +CoreferenceResolver +References +"the company" → Apple Inc. + + + + + + + + + +TripletExtractor — (subject, predicate, object) +(Steve Jobs, founded, Apple Inc.) · (Apple Inc., located_in, Cupertino) + diff --git a/docs/assets/img/diagrams/graphrag-flow.svg b/docs/assets/img/diagrams/graphrag-flow.svg new file mode 100644 index 00000000..1446775e --- /dev/null +++ b/docs/assets/img/diagrams/graphrag-flow.svg @@ -0,0 +1,100 @@ + +GraphRAG Retrieval Flow + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +GraphRAG — Graph-Augmented Retrieval Flow + + + + +QUERY +User Query +"Who founded +Apple Inc.?" + + + + + + + + +Vector Search +Embedding similarity +top-k semantic matches + + + + +Graph Traversal +Relation paths + context +entity neighborhood + + + + + + + + +CONTEXT +Build Context +Merge vectors + graph +rank by relevance +attach source links +provenance metadata + + + + + + + +GENERATE +LLM +Grounded generation +context-aware answer +no hallucination + + + + + + + +ANSWER +Output +Traceable +answer ++ source nodes + + +Every claim links back to a source node — no black-box outputs + diff --git a/docs/assets/img/diagrams/kg-structure.svg b/docs/assets/img/diagrams/kg-structure.svg new file mode 100644 index 00000000..47e6b659 --- /dev/null +++ b/docs/assets/img/diagrams/kg-structure.svg @@ -0,0 +1,90 @@ + +Knowledge Graph Structure + + + + + + + + + + + + + +Knowledge Graph — Entity & Relation Structure + + + + + + +founded + + + + +connected_to + + + + +located_in + + + + +born_in + + + + + + + + + + +Steve Jobs +PERSON + + + + +Apple Inc. +ORGANIZATION + + + + +Cupertino +LOCATION + + + + +1976 +DATE + + + + +PERSON + + + +ORGANIZATION + + + +LOCATION + + + +DATE + + + +inferred relation + diff --git a/docs/assets/img/diagrams/pipeline-flow.svg b/docs/assets/img/diagrams/pipeline-flow.svg new file mode 100644 index 00000000..a8038810 --- /dev/null +++ b/docs/assets/img/diagrams/pipeline-flow.svg @@ -0,0 +1,85 @@ + +Semantica Pipeline Flow + + + + + + + + + + + + + + + + +01 +Ingest +FileIngestor + + + + + + +02 +Parse +DocumentParser + + + + + + +03 +Normalize +TextNormalizer + + + + + + +04 +Extract +NERExtractor + + + + + + +05 +Build KG +GraphBuilder + + + + + + +06 +QA +ConflictDetector + + + + + + +07 +Store +VectorStore + + + + + + +08 +Deliver +AgentContext + diff --git a/docs/assets/img/diagrams/reasoning-chain.svg b/docs/assets/img/diagrams/reasoning-chain.svg new file mode 100644 index 00000000..f482daec --- /dev/null +++ b/docs/assets/img/diagrams/reasoning-chain.svg @@ -0,0 +1,90 @@ + +Reasoning and Inference Chain + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Reasoning Engine — Forward Chaining Inference + + + + +KNOWN FACT +Steve Jobs +founded Apple Inc. + + + + +KNOWN FACT +Apple Inc. +headquartered in Cupertino + + + + + + +Base Facts + + + + +RULE ENGINE +IF / THEN Rule +IF ?x founded ?y +AND ?y located_in ?z +THEN ?x connected_to ?z + + + + + + + + + +DERIVED FACT +Steve Jobs connected_to Cupertino +confidence: 0.91 · derived via rule #3 + + + + +EXPLANATION PATH +Jobs→founded→Apple→located_in→Cupertino +traceable steps · not a black box + + +Derived Knowledge + + +Rete · Forward Chain · Datalog + + +Every derived fact ships with a traceable explanation path — zero black-box conclusions + diff --git a/docs/assets/img/semantica-wordmark-dark.svg b/docs/assets/img/semantica-wordmark-dark.svg new file mode 100644 index 00000000..ba58497b --- /dev/null +++ b/docs/assets/img/semantica-wordmark-dark.svg @@ -0,0 +1,12 @@ + + Semantica + + + S + + Semantica + diff --git a/docs/assets/img/semantica-wordmark-light.svg b/docs/assets/img/semantica-wordmark-light.svg new file mode 100644 index 00000000..bc207671 --- /dev/null +++ b/docs/assets/img/semantica-wordmark-light.svg @@ -0,0 +1,12 @@ + + Semantica + + + S + + Semantica + diff --git a/docs/concepts.md b/docs/concepts.md index 88eaa595..107d6059 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -14,6 +14,8 @@ At its core, Semantica adds a **context and intelligence layer** on top of your ## Knowledge Graphs +Knowledge graph node and edge structure showing entities (Person, Organization, Location, Date) and their typed relations + The foundation of everything in Semantica. A knowledge graph stores information as three building blocks: - **Nodes (entities)** — people, companies, locations, events, concepts @@ -76,6 +78,8 @@ Semantica uses embeddings for: GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses by grounding them in a structured knowledge graph rather than raw text chunks alone. +GraphRAG flow: User Query → Vector Search + Graph Traversal → Context Builder → LLM → Grounded Answer + **How it works:** 1. User submits a query diff --git a/docs/docs.json b/docs/docs.json index c2ffe747..a1813096 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -10,8 +10,8 @@ }, "favicon": "/assets/img/semantica-logo.png", "logo": { - "light": "/assets/img/semantica-logo.png", - "dark": "/assets/img/semantica-logo.png" + "light": "/assets/img/semantica-wordmark-light.svg", + "dark": "/assets/img/semantica-wordmark-dark.svg" }, "font": { "headings": { diff --git a/docs/index.md b/docs/index.md index d4845ac2..ec684db1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -34,6 +34,8 @@ Semantica is the **accountability and context layer** you add on top of your exi Works alongside any LLM provider and any agent framework. +Semantica four-layer architecture: Ingestion → Processing → Intelligence → Application + ## Quick Start ```bash diff --git a/docs/quickstart.md b/docs/quickstart.md index ae68c050..dfb11686 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -39,6 +39,8 @@ python -c "import semantica; print(semantica.__version__)" ## Full Pipeline +Semantica end-to-end pipeline: Ingest → Parse → Normalize → Extract → Build KG → QA → Store → Deliver + diff --git a/docs/reference/context.md b/docs/reference/context.md index 02b7d411..44042e70 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -15,6 +15,8 @@ icon: "brain" - **`CausalAnalyzer`** — traces downstream impact of any decision - **`PolicyEngine`** — validates decisions against configurable rules before they're recorded +AgentContext hub: AI Agent calls store/retrieve against VectorStore and record_decision against ContextGraph + ## AgentContext The main entry point. Wraps memory, graph, and decision tracking behind a single API. diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 09d4d327..f823bf6d 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -21,6 +21,8 @@ icon: "diagram-project" For conflict detection and advanced entity resolution, use `semantica.conflicts` and `semantica.deduplication` alongside this module. +Knowledge graph entity and relation structure: Person, Organization, Location, Date nodes with typed labeled edges + ## GraphBuilder Constructs knowledge graphs from extracted entities and relationships: diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index be9c7d9b..32723a7f 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -14,6 +14,8 @@ icon: "gear" - **`FailureHandler`** — skip, stop, or retry failed documents without halting the pipeline - **Progress tracking** — console (tqdm), WebSocket streaming, or file logging +Pipeline step sequence: Ingest → Parse → Normalize → Extract → Build KG → QA → Store → Deliver + ## Basic Pipeline ```python diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 9012abbb..876e67b8 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -16,6 +16,8 @@ icon: "microchip" - **`TemporalReasoningEngine`** — all 13 Allen interval algebra relations for time-aware inference - **`ExplanationGenerator`** — structured explanation paths for every derived conclusion +Forward chaining inference: known facts + IF/THEN rules produce derived facts with a full traceable explanation path + ## Reasoner (Main Facade) The unified entry point for rule-based forward-chaining inference: diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index b3f1cd18..63ff85ae 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -14,6 +14,8 @@ icon: "magnifying-glass-chart" - **`EventExtractor`** — event detection with participants, temporal context, and confidence scores - **`CoreferenceResolver`** — resolve "Apple" and "the company" to the same entity across a document +Semantic extraction pipeline: raw text fans into NER, Relation, and Coreference extractors, then merges into a Triplet Generator + ## NERExtractor ```python