mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
llms.md: - Only Groq/OpenAI/LiteLLM/HuggingFaceLLM are exported — remove non-exported Anthropic/Ollama/Gemini/DeepSeek/Novita as direct imports - Rename HuggingFace -> HuggingFaceLLM (correct class name) - Remove non-existent create_provider() — replace with LiteLLM provider/model pattern - Add LiteLLM 100+ providers section with provider/model string examples - Add Exported Classes table (class -> provider -> API key) - Update Provider Comparison table to show correct import per provider ontology.md: - Remove non-existent OntologyManager — replace with OntologyEngine facade - Remove non-existent start_explorer() — replace with CLI: semantica-explorer - SHACLValidator -> OntologyValidator (correct exported name) - OWLExporter -> OWLGenerator (correct exported name) - Add Exported Classes block with all 15+ exported symbols - Add LLMOntologyGenerator section, NamespaceManager section - Add OntologyEvaluator section with coverage/completeness metrics - Add ingest_ontology() section - Add versioning moved-to note (change_management module) kg.md: - TemporalKnowledgeGraph does not exist — replace with TemporalGraphQuery - DistanceCalculator does not exist — replace with SimilarityCalculator - Add Exported Classes block with all 20+ exported symbols - Fix temporal example to use TemporalGraphQuery + TemporalVersionManager correctly - Add SimilarityCalculator section with NodeEmbedder integration example provenance.md: - ActivityTracker not exported — remove; ProvenanceManager handles tracking - Fix track_entity() signature: add source_location, source_quote params - Fix GraphBuilderWithProvenance import: from semantica.kg, not semantica.provenance - Add Exported Classes block with storage backends and checksum utilities - Add SourceReference section with DOI/page/quote fields - Add tamper-evident checksum section (compute_checksum/verify_checksum) - Add Enable Provenance in Extractors section - Fix duplicate heading (W3C PROV-O Export appeared twice) reasoning.md: - Add Exported Classes block with all engines + data types + explanation types - Add Quick Start section - Add Choosing an Engine comparison table - Add InferenceResult/Explanation/ReasoningStep type annotations in examples - Add Tip: use DatalogReasoner for recursive rules semantic_extract.md: - Add Exported Classes block with NamedEntityRecognizer, EventDetector, Entity, Relation, Event, CoreferenceChain, EntityClassifier, TemporalEventProcessor - Add Quick Start section (one-liner extraction pipeline) - Rename EventExtractor -> EventDetector (correct exported name) - Clarify NERExtractor vs NamedEntityRecognizer distinction - Add return type annotations to EventDetector example core.md: - Add Exported Classes block - Add When to Use Core vs. Individual Modules decision table - Add Tip: LifecycleManager only for long-running apps - Fix MethodRegistry example to import build_knowledge_base correctly parse.md: - Add Exported Classes block with all format-specific parsers + data types - Add DoclingParser optional import note utils.md: - Add Exported Classes block with logging/validation/progress/helpers/exceptions deduplication.md: - Add Exported Classes block with PropertyMergeRule, MergeStrategyManager, method_registry, and all convenience functions export.md: - Add Exported Classes block with all exporters, NamespaceManager, SemanticNetworkYAMLExporter, and all convenience functions
164 lines
5.3 KiB
Markdown
164 lines
5.3 KiB
Markdown
---
|
|
title: "Parse Module"
|
|
description: "Document parsing and text extraction — DocumentParser for standard formats and DoclingParser for complex layouts."
|
|
icon: "file-lines"
|
|
---
|
|
|
|
`semantica.parse` extracts structured text, layout, tables, and metadata from unstructured documents. `DocumentParser` handles clean machine-readable files; `DoclingParser` handles complex layouts, scanned PDFs, and multi-column documents.
|
|
|
|
## Exported Classes
|
|
|
|
```python
|
|
from semantica.parse import (
|
|
DocumentParser, # auto-detect format — delegates to format-specific parser
|
|
PDFParser, # PDF text extraction
|
|
DOCXParser, # Word .docx documents
|
|
HTMLParser, # HTML / web pages
|
|
MarkdownParser, # Markdown files
|
|
TXTParser, # plain text
|
|
JSONParser, # JSON documents
|
|
XMLParser, # XML documents
|
|
CSVParser, # CSV / TSV files
|
|
WebParser, # URL fetch + HTML parsing
|
|
EmailParser, # .eml / .msg email files
|
|
CodeParser, # source code files
|
|
# Data types
|
|
ParsedDocument, # {text, sections, tables, metadata, source_id}
|
|
DocumentMetadata, # {title, author, created_date, page_count, language, ...}
|
|
)
|
|
|
|
# Optional — requires: pip install "semantica[docling]"
|
|
from semantica.parse import DoclingParser # advanced OCR + layout analysis
|
|
```
|
|
|
|
## What You Get
|
|
|
|
- **`DocumentParser`** — standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX — auto-detects format
|
|
- **`DoclingParser`** — advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR (optional dep)
|
|
- **`ParsedDocument`** — structured output with `text`, `sections`, `tables`, and `metadata`
|
|
- **Format-specific parsers** — `PDFParser`, `DOCXParser`, `HTMLParser`, `WebParser`, `EmailParser`, `CodeParser`, etc.
|
|
|
|
## DocumentParser
|
|
|
|
Standard parser for clean, machine-readable documents:
|
|
|
|
```python
|
|
from semantica.parse import DocumentParser
|
|
|
|
parser = DocumentParser()
|
|
parsed = parser.parse("data/report.pdf")
|
|
|
|
print(parsed.text) # full clean text
|
|
print(parsed.metadata) # title, author, date, page_count, language, etc.
|
|
print(parsed.sections) # document structure as a list of Section objects
|
|
```
|
|
|
|
Supported formats: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX.
|
|
|
|
## DoclingParser
|
|
|
|
Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot:
|
|
|
|
```bash
|
|
pip install "semantica[docling]"
|
|
```
|
|
|
|
```python
|
|
from semantica.parse import DoclingParser
|
|
|
|
parser = DoclingParser(
|
|
extract_tables=True, # structured table extraction with cell type detection
|
|
extract_images=True, # extract image regions for downstream OCR
|
|
output_format="markdown", # "markdown" | "html" | "json"
|
|
)
|
|
|
|
parsed = parser.parse("data/annual_report.pdf")
|
|
|
|
print(parsed.text) # full clean text
|
|
print(parsed.tables) # structured TableData objects with headers and rows
|
|
print(parsed.sections) # document structure with heading hierarchy
|
|
```
|
|
|
|
Use `DoclingParser` for:
|
|
|
|
- Multi-column PDF layouts
|
|
- Tables with merged cells or complex headers
|
|
- PPTX slides with embedded charts
|
|
- XLSX spreadsheets with formulas
|
|
- Scanned documents with OCR
|
|
- Academic papers and technical reports
|
|
|
|
## OCR Support
|
|
|
|
```python
|
|
parser = DoclingParser(
|
|
ocr=True,
|
|
ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents
|
|
extract_tables=True,
|
|
)
|
|
|
|
parsed = parser.parse("data/scanned_contract.pdf")
|
|
```
|
|
|
|
## Parsed Document Object
|
|
|
|
Both parsers return a `ParsedDocument` with the same structure:
|
|
|
|
```python
|
|
@dataclass
|
|
class ParsedDocument:
|
|
text: str # full extracted text
|
|
sections: List[Section] # heading-based document structure
|
|
tables: List[TableData] # structured table data (DoclingParser only)
|
|
metadata: DocumentMetadata # title, author, dates, page count
|
|
source_id: str # links back to the original DataSource
|
|
|
|
@dataclass
|
|
class DocumentMetadata:
|
|
title: Optional[str]
|
|
author: Optional[str]
|
|
created_date: Optional[datetime]
|
|
page_count: int
|
|
language: Optional[str] # ISO 639-1 code
|
|
has_tables: bool
|
|
has_images: bool
|
|
word_count: int
|
|
format: str # "pdf" | "docx" | "pptx" | ...
|
|
```
|
|
|
|
## Integration with FileIngestor
|
|
|
|
The most common pattern — ingest a directory then parse each source:
|
|
|
|
```python
|
|
from semantica.ingest import FileIngestor
|
|
from semantica.parse import DoclingParser
|
|
|
|
ingestor = FileIngestor()
|
|
parser = DoclingParser(extract_tables=True)
|
|
|
|
sources = ingestor.ingest("data/reports/")
|
|
for source in sources:
|
|
parsed = parser.parse(source)
|
|
# → parsed.text, parsed.tables, parsed.sections
|
|
```
|
|
|
|
<Note>
|
|
Docling is an optional dependency. If `docling` is not installed, `DoclingParser` raises an `ImportError` with installation instructions. `DocumentParser` is always available and requires no extras.
|
|
</Note>
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Ingest" icon="database" href="ingest">
|
|
Load files before parsing.
|
|
</Card>
|
|
<Card title="Split" icon="scissors" href="split">
|
|
Chunk parsed text for embedding and extraction.
|
|
</Card>
|
|
<Card title="Docling Integration" icon="file-pdf" href="../integrations/docling">
|
|
Full Docling integration setup guide.
|
|
</Card>
|
|
<Card title="Semantic Extract" icon="magnifying-glass" href="semantic_extract">
|
|
Extract entities and relations from parsed text.
|
|
</Card>
|
|
</CardGroup>
|