Files
semantica/tests
Mohd KaifandKaifAhmad1 aa712b9110 feat: implement Apache Arrow and Feather file ingestion support (#235) (#705)
* feat: implement Apache Arrow and Feather file ingestion support (#235)

* fix(arrow): eliminate double full-scan and clean up reader wrapper

- Replace _read_batches with _read_batches_with_info which collects
  batch metadata (total_rows, record_batches) during the same pass as
  the data read, so ingest_file no longer calls _file_metadata before
  _read_batches. For a limit=1 read on a large file this previously
  scanned every batch twice; now it stops after the first batch.

- _file_metadata is now only invoked for include_data=False (where a
  full scan is unavoidable to report accurate row counts).

- Remove the dead num_record_batches property from _ArrowReaderWrapper;
  it was never called by production code and its is_table branch
  materialised all batches just to count them.

- Fix _open_file exception chain: raise ... from file_err instead of
  from feather_err so the most diagnostic IPC error appears in the
  Python traceback chain, not the least informative fallback error.

* docs(changelog): add [Unreleased] entries for Arrow ingestion (#705)

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-06-28 12:57:17 +05:30
..
2026-06-16 04:43:45 +05:00