fix: prevent fallback recursion and write proper Parquet in embed generate command

Fixes #994:
1. Prevent self-recursion in methods.py: generate_embeddings, embed_text,
   calculate_similarity, and pool_embeddings all registered themselves as
   custom methods, causing infinite self-calls when dispatch invoked them
   without explicitly passing method parameter.
   Fix: check custom_method is not the function itself before recursing.

2. Fix embed generate --output corrupt output: the CLI wrote
   json.dumps(result, default=str) which produced plaintext repr of numpy
   arrays (e.g. '[1.49e-01 4.85e-02 ...]') instead of proper Parquet.
   Fix: detect .parquet extension (case-insensitive), convert numpy array
   to pandas DataFrame with dim_* columns and id index, use to_parquet().
   Non-parquet extensions fall back to JSON with clear ImportError message.

3. Add pyarrow>=14.0.0 to core dependencies (previously only in
   ingest-parquet/ingest-arrow optional extras). The documented quick-start
   flow of embed generate --output ... requires pyarrow out of the box.
   (Note: pandas>=1.3.0 is already a core dependency; pyarrow is the
   missing piece.)

Note: .github/workflows/* files are excluded from this PR as they require
a token with workflow scope. Upstream workflows are unchanged.
This commit is contained in:
Varun Sahni
2026-08-15 11:52:04 +05:30
parent c1be6dd7dc
commit d42af280e8
12 changed files with 32 additions and 779 deletions
+2 -1
View File
@@ -85,7 +85,8 @@ dependencies = [
"loguru>=0.7.3",
"structlog>=22.1.0",
"gensim>=4.4.0",
"httpx<0.29.0"
"httpx<0.29.0",
"pyarrow>=14.0.0"
]
[project.urls]