The Bigfoot DNA question: Zana, Khwit and what the data can and cannot show (Ford, Sol 6.0, Claude Opus 5/5.5)

Ah — then my ~2.97 GB expectation was wrong. I likely carried over Claude’s 2.974× coverage number and/or stale file-size metadata as though it were the BAM size.

If VK6.final.bam is actually close to 10 GB, trust the file on disk, not that earlier estimate. That also means the “8 GB total” control-download estimate is wrong and the whole batch may be much larger.

Let’s get the exact size and see whether curl is still growing it:

ls -lh /Volumes/AI/genomics/controls/
stat -f "%N  %z bytes" /Volumes/AI/genomics/controls/VK6.final.bam
ps -axo pid,etime,comm,args | grep -E 'VK6|VK87|VK548' | grep -v grep

Paste that and I’ll recalculate where we actually are.

Found it — and the error was mine, not the download's. My sizes came from NCBI's size_MB, which is the SRA archive object size, not the submitted BAM. The real files are ~3.2× larger.

Not done. And I need to correct a figure I gave you.

The md5 failed for VK6 — the checksum caught real corruption from the earlier resume. Let me stop, and test whether parallel range requests can fix the 0.9 MB/s crawl.

Fair catch on the unquoted glob. Using literal paths.

3× faster and byte-exact. Implementing chunked parallel fetch with per-chunk retries — which also fixes the corruption cause, since no whole-file resume is involved.

No — and two things went wrong that you should know about.

I gave you a wrong number

The core download is 27.1 GiB, not 8.00 GB. My sizes came from NCBI SRA runinfo's size_MB, which is the size of the SRA archive object, not the submitted BAM. It understates by ~3.2×. Real sizes, from the data host's own Content-Length:

The error entered because ENA's portal API — which serves the correct submitted_bytes field, and which I used for Zana/Khwit — was down, so I fell back to NCBI and conflated the two fields. The manifest is corrected and the script now takes sizes from the host rather than trusting any metadata field. You authorized 8 GB; the real cost is 3.4× that. Disk is irrelevant (2.5 TiB free), but the time isn't.

The first VK6 download was corrupt

It reached the exact right byte count and failed its MD5. Cause: curl -C - whole-file resume across retries. The checksum caught it, which is the one piece of good news — correctness was never at risk, only my ETA.

I misdiagnosed this at first. Seeing 9,548 MB against an expected 2,965 MB, I assumed a runaway append and killed it. The file size was actually right and my expectation was wrong. The corruption was real, but not for the reason I guessed.

Fixed properly

I replaced the shell fetcher with a chunked parallel downloader:

• Range requests, 8 streams, 64 MB chunks. Benchmarked: 3.33 MB/s vs 1.13 MB/s single-stream, and I verified parallel reassembly is byte-identical to a single stream.
• Per-chunk retries with length checks. A short read is rejected rather than appended, so the whole-file-resume failure mode is gone by construction.
• Completed chunks survive a restart, so an interrupted run doesn't start over.
• MD5 still verified on the assembled file, then samtools index since ENA publishes no .bai.

Where it stands

VK6 is at 16 of 150 chunks, 671 MB. At ~3.3 MB/s the full 27.1 GiB takes roughly 2.3 hours — rather than the 8.5 it would have taken at the old rate.

Nothing about the experiment changed: same three authorized BAMs, same frozen pipeline, --bam and --skip-x still predeclared, no threshold touched. Phase 3 locked, Bigfoot unpublished.

One thing worth your call: given that 27 GiB is well past the figure you approved, say if you'd rather I drop to two genomes (VK6 + VK87, ~17.9 GiB) and lose the third pairwise comparison. I'll keep all three running unless you say otherwise.

VK6 failed md5 again, with a different hash than the first attempt. Two different download methods, two different wrong hashes — that needs diagnosing before more hours burn.

BAM is structurally valid and reproduces the served bytes. Let me test the boundaries and the read count — if local matches served, then ENA's recorded checksum is the odd one out.

My dd offsets weren't MB-aligned, so three of those comparisons were invalid — my test bug, not the file's. Redoing with exact byte seeks.