fetch: 'builder error' on image-sibling + parquet-only datasets #84

New issue

Open

opened 2026-07-27 07:46:57 +00:00 by vxfemboy · 0 comments

vxfemboy commented

2026-07-27 07:46:57 +00:00

Owner

`kibble fetch` "builder error" on datasets with image siblings or parquet-only layouts

Some HF datasets fail with download failed for <id>: builder error while others (single .jsonl) succeed. Reproduced building the kitby-v1 corpus:

ianncity/GLM-5.2-Conversation — a single dataset.jsonl plus a .webp sibling → builder error (had to direct-download the jsonl).
r0b0tlab/qwen3.8-max-distillation-50k — parquet shards under data/ (no top-level jsonl) → builder error.
SupraLabs/reasoning-corpus-4K-5M-v1 (single jsonl) and greghavens/kimi-k3-... (jsonl) → succeed.

The message text (download failed for {url}: {e} with {url} = the bare dataset id, {e} = builder error) points at a bare id reaching reqwest as a URL on the parquet-convert / fallback path — a RequestBuilder build failure, not an HTTP error. is_data_file already accepts parquet, so the parquet case should resolve via data/ or the parquet-convert branch; something in that path constructs the request from the id instead of a resolve URL.

Impact: common dataset layouts (image sibling; parquet-only) need a manual direct-download workaround.

Fix direction: ensure every fetch path builds a full resolve/main/... (or /parquet) URL, never the bare id; and skip non-data siblings (.webp/.png/...) before the builder step.

Found during kitby-v1 data foundry (P1).

## `kibble fetch` "builder error" on datasets with image siblings or parquet-only layouts Some HF datasets fail with `download failed for <id>: builder error` while others (single `.jsonl`) succeed. Reproduced building the kitby-v1 corpus: - `ianncity/GLM-5.2-Conversation` — a single `dataset.jsonl` **plus a `.webp`** sibling → builder error (had to direct-download the jsonl). - `r0b0tlab/qwen3.8-max-distillation-50k` — **parquet shards under `data/`** (no top-level jsonl) → builder error. - `SupraLabs/reasoning-corpus-4K-5M-v1` (single jsonl) and `greghavens/kimi-k3-...` (jsonl) → **succeed**. The message text (`download failed for {url}: {e}` with `{url}` = the bare dataset **id**, `{e}` = `builder error`) points at a bare id reaching `reqwest` as a URL on the parquet-convert / fallback path — a `RequestBuilder` build failure, not an HTTP error. `is_data_file` already accepts parquet, so the parquet case should resolve via `data/` or the parquet-convert branch; something in that path constructs the request from the id instead of a resolve URL. **Impact:** common dataset layouts (image sibling; parquet-only) need a manual direct-download workaround. **Fix direction:** ensure every fetch path builds a full `resolve/main/...` (or `/parquet`) URL, never the bare id; and skip non-data siblings (`.webp/.png/...`) before the builder step. Found during kitby-v1 data foundry (P1).