Fsspec with block granularity - #701

Merged

FrancescAlted merged 46 commits into

mainfrom

Aug 17, 2026

Commits on Aug 16, 2026

Add a plan for reading Blosc2 containers through fsspec URLs

FrancescAlted
and
claude
committed
9dc41b8
Read and write Blosc2 containers through fsspec URLs

FrancescAlted
and
claude
committed
1ee2f32
Open remote containers through a local fsspec cache

FrancescAlted
and
claude
committed
2a5dd2c
Read remote frames chunk by chunk with lazy=True

FrancescAlted
and
claude
committed
afbdcea
Name the http test after what it checks

FrancescAlted
and
claude
committed
f5b4ba2
Let a Proxy pick up the cache left by an earlier run

FrancescAlted
and
claude
committed
9ba346e
Keep the install page about installing

FrancescAlted
and
claude
committed
c5961c5
Trim the fsspec docs to one home per fact

FrancescAlted
and
claude
committed
ea7543b
Let lazy= and cache_storage= compose

FrancescAlted
and
claude
committed
7e6d6e6
Stamp the chunk cache with fsspec's ukey, not guessed metadata

FrancescAlted
and
claude
committed
01ca574
Save a container to an fsspec URL in one PUT

FrancescAlted
and
claude
committed
950d3bf
Add an fsspec read/write example

FrancescAlted
and
claude
committed
2da443f
Point the fsspec docs at the runnable example

FrancescAlted
and
claude
committed
f305665
Fetch a remote chunk in one range read, statelessly

FrancescAlted
and
claude
committed
215b6b7
Overlap chunk fetches in Proxy.fetch with a thread pool

FrancescAlted
and
claude
committed
77f4304
Default a lazy fsspec proxy to 8 concurrent fetches

FrancescAlted
and
claude
committed
5beb7ef
Add a concurrent-fetch example for lazy fsspec arrays

FrancescAlted
and
claude
committed
be00ee8
Link the concurrency example from the docs

FrancescAlted
and
claude
committed
8e4364c
Bring the plan up to date with what phase 3 became

FrancescAlted
and
claude
committed
c7f916d
Install fsspec in the test group so CI actually runs its tests

FrancescAlted
and
claude
committed
1fd52c2
Add a benchmark for the concurrency default against a real endpoint

FrancescAlted
and
claude
committed
2cc1f12
Document the moto[server] recipe in the benchmark

FrancescAlted
and
claude
committed
090e149
Fix aget_chunk against real async filesystems

FrancescAlted
and
claude
committed
01dfef5
Test the fsspec support against a real S3 endpoint

FrancescAlted
and
claude
committed
42fac93
Address the PR review: file:// URLs, cache identity, cache geometry

FrancescAlted
and
claude
committed
d2fe249
Fix six defects found reviewing the frame reader

FrancescAlted
and
claude
committed
ea09cb8
Fix what two reviews found in the fsspec support

FrancescAlted
and
claude
committed
5aec265
Check the source identity when a Proxy adopts a cache

FrancescAlted
and
claude
committed
bf87015
Track fetched chunks explicitly in the Proxy cache

FrancescAlted
and
claude
committed
ff47bae
Fix file:// drive URLs on POSIX and .b2d detection with a query
FrancescAlted
committed
2b3a112
Trim the fsspec proxy plumbing
FrancescAlted
committed
82a971a
Snapshot the sidecar caches before iterating them
FrancescAlted
committed
c6285e1
Fetch blocks, not whole chunks, from a lazy fsspec array

FrancescAlted
and
claude
committed
473511f
Say what lazy=True fetches now that it is not whole chunks

FrancescAlted
and
claude
committed
449d459
Stop truncating lazy chunks down to their header

FrancescAlted
and
claude
committed
aacd9cc
Analyse and measure block-granular fsspec downloads

FrancescAlted
and
claude
committed
6ff74e2
Benchmark block fetching end to end, against a moto S3 server

FrancescAlted
and
claude
committed
4e1db6c
Record what route A became

FrancescAlted
and
claude
committed
d434ad2

Commits on Aug 17, 2026

Note the rewrite-per-fetch ceiling where it lives

FrancescAlted
and
claude
committed
1f569f0
Stop overselling persisted block layouts

FrancescAlted
and
claude
committed
66f9bfe
Stop reading a chunk back to find out what is already in it

FrancescAlted
and
claude
committed
cbe577e
Record what the route B prototype settled

FrancescAlted
and
claude
committed
91d5e75
Finish waiting for modal screens instead of pausing once

FrancescAlted
and
claude
committed
161b008
Address the review: keep the proxy's bookkeeping its own

FrancescAlted
and
claude
committed
372dc28
Address the review: chunks that cannot be spliced, and evictions that outlive a proxy

FrancescAlted
and
claude
committed
dbf6cdf
Do not read block offsets out of a chunk that has none

FrancescAlted
and
claude
committed
484f502