mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
* docs: replace Exported Classes import blocks with summary tables across all 25 modules * docs: add method/parameter tables to parse, ingest, ontology, normalize, triplet_store, change_management, conflicts, export, graph_store, provenance, and semantic_extract modules
6.1 KiB
6.1 KiB
title, description, icon
| title | description | icon |
|---|---|---|
| Parse Module | Document parsing and text extraction — DocumentParser for standard formats and DoclingParser for complex layouts. | file-lines |
semantica.parse extracts structured text, layout, tables, and metadata from unstructured documents. DocumentParser handles clean machine-readable files; DoclingParser handles complex layouts, scanned PDFs, and multi-column documents.
Exported Classes
| Class | Role |
|---|---|
DocumentParser |
Auto-detects format — delegates to format-specific parser (PDF, DOCX, HTML, JSON, CSV, ...) |
DoclingParser |
Complex layouts, merged-cell tables, multi-column PDFs, and OCR (pip install semantica[docling]) |
ParsedDocument |
{text, sections, tables, metadata, source_id} — structured output from any parser |
DocumentMetadata |
{title, author, created_date, page_count, language, word_count} |
PDFParser |
PDF text and metadata extraction |
WebParser |
URL fetch + HTML parsing |
EmailParser |
.eml / .msg email files with attachment extraction |
CodeParser |
Source code files with syntax-aware block detection |
DocumentParser
Standard parser for clean, machine-readable documents:
from semantica.parse import DocumentParser
parser = DocumentParser()
parsed = parser.parse("data/report.pdf")
print(parsed.text) # full clean text
print(parsed.metadata) # title, author, date, page_count, language, etc.
print(parsed.sections) # document structure as a list of Section objects
Supported formats: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX.
DoclingParser
Advanced parser using the Docling backend — handles layouts that DocumentParser cannot:
pip install "semantica[docling]"
from semantica.parse import DoclingParser
parser = DoclingParser(
extract_tables=True, # structured table extraction with cell type detection
extract_images=True, # extract image regions for downstream OCR
output_format="markdown", # "markdown" | "html" | "json"
)
parsed = parser.parse("data/annual_report.pdf")
print(parsed.text) # full clean text
print(parsed.tables) # structured TableData objects with headers and rows
print(parsed.sections) # document structure with heading hierarchy
Use DoclingParser for:
- Multi-column PDF layouts
- Tables with merged cells or complex headers
- PPTX slides with embedded charts
- XLSX spreadsheets with formulas
- Scanned documents with OCR
- Academic papers and technical reports
OCR Support
parser = DoclingParser(
ocr=True,
ocr_language=["en"], # ISO 639-1 codes; list for multi-language documents
extract_tables=True,
)
parsed = parser.parse("data/scanned_contract.pdf")
Supported Formats
| Format | Extension | Parser Used | Notes |
|---|---|---|---|
.pdf |
PDFParser / DoclingParser |
Text, tables, metadata; Docling adds OCR | |
| Word | .docx |
Built-in | Text, headings, tables, metadata |
| HTML | .html, .htm |
HTMLParser / WebParser |
WebParser fetches remote URLs |
| Markdown | .md |
Built-in | Preserves heading hierarchy |
| Plain text | .txt |
TXTParser |
Minimal metadata |
| JSON | .json |
JSONParser |
One object per line or array |
| CSV / TSV | .csv, .tsv |
CSVParser |
Header auto-detected |
| Excel | .xlsx, .xls |
Built-in | Sheet selection supported |
| PowerPoint | .pptx |
Built-in | DoclingParser for embedded charts |
.eml, .msg |
EmailParser |
Attachments extracted | |
| XML | .xml |
XMLIngestor |
XXE-safe, optional XSD validation |
| Archive | .zip, .tar |
FileIngestor |
Recursive extraction |
| Source code | .py, .js, .java, ... |
CodeParser |
AST-aware block detection |
Parsed Document Object
Both parsers return a ParsedDocument with the same structure:
@dataclass
class ParsedDocument:
text: str # full extracted text
sections: List[Section] # heading-based document structure
tables: List[TableData] # structured table data (DoclingParser only)
metadata: DocumentMetadata # title, author, dates, page count
source_id: str # links back to the original DataSource
@dataclass
class DocumentMetadata:
title: Optional[str]
author: Optional[str]
created_date: Optional[datetime]
page_count: int
language: Optional[str] # ISO 639-1 code
has_tables: bool
has_images: bool
word_count: int
format: str # "pdf" | "docx" | "pptx" | ...
DocumentParser Methods
| Method | Returns | Description |
|---|---|---|
parse(source) |
ParsedDocument |
Auto-detect format and extract text, sections, metadata |
parse_batch(sources) |
List[ParsedDocument] |
Process multiple sources in parallel |
is_supported(path) |
bool |
Check if the file extension is supported |
Integration with FileIngestor
The most common pattern — ingest a directory then parse each source:
from semantica.ingest import FileIngestor
from semantica.parse import DoclingParser
ingestor = FileIngestor()
parser = DoclingParser(extract_tables=True)
sources = ingestor.ingest("data/reports/")
for source in sources:
parsed = parser.parse(source)
# → parsed.text, parsed.tables, parsed.sections