mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: implement Apache Arrow and Feather file ingestion support (#235) (#705)

Browse Source 
* feat: implement Apache Arrow and Feather file ingestion support (#235)

* fix(arrow): eliminate double full-scan and clean up reader wrapper

- Replace _read_batches with _read_batches_with_info which collects
  batch metadata (total_rows, record_batches) during the same pass as
  the data read, so ingest_file no longer calls _file_metadata before
  _read_batches. For a limit=1 read on a large file this previously
  scanned every batch twice; now it stops after the first batch.

- _file_metadata is now only invoked for include_data=False (where a
  full scan is unavoidable to report accurate row counts).

- Remove the dead num_record_batches property from _ArrowReaderWrapper;
  it was never called by production code and its is_table branch
  materialised all batches just to count them.

- Fix _open_file exception chain: raise ... from file_err instead of
  from feather_err so the most diagnostic IPC error appears in the
  Python traceback chain, not the least informative fallback error.

* docs(changelog): add [Unreleased] entries for Arrow ingestion (#705)

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
This commit is contained in:
Mohd Kaif
2026-06-28 12:57:17 +05:30
committed by GitHub
co-authored by KaifAhmad1
parent c93af4a514 f2c60256f1
commit aa712b9110
9 changed files with 1200 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pyproject.toml
+3 -2
View File
@@ -120,6 +120,7 @@ parse-docling = ["docling>=1.0.0"]
db-snowflake = ["snowflake-connector-python>=4.5.0", "cryptography>=49.0.0"]
db-arrow = ["pyarrow>=21.0.0"]
ingest-parquet = ["pyarrow>=21.0.0"]
ingest-arrow = ["pyarrow>=21.0.0"]
db-all = [
"semantica[db-snowflake,db-arrow]"
@@ -232,8 +233,8 @@ explorer-lite = [
# Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
all = [
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,agno]"
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,agno]"
]
# ---------------- ENTRYPOINTS ----------------