Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
c15bda7
Add the plan for block-granular Caterva2 reads
FrancescAlted Aug 17, 2026
7bb0eff
Pool the HTTP client C2Array requests go through
FrancescAlted Aug 17, 2026
422ba4b
Lift the frame reading out of FsspecNDSource into ByteRangeNDSource
FrancescAlted Aug 17, 2026
d541eb3
Read a C2Array's blocks over HTTP ranges
FrancescAlted Aug 17, 2026
b021104
Ask a Caterva2 subscriber for a whole wave of ranges at once
FrancescAlted Aug 17, 2026
1b2dd44
Record what the block-granularity work landed as, and what it found
FrancescAlted Aug 17, 2026
b868c53
Keep C2NDSource's max_ranges with the rest of its class body
FrancescAlted Aug 17, 2026
e8d9ba1
Measure the Caterva2 block path the way the fsspec one is measured
FrancescAlted Aug 18, 2026
f9f755c
Say when a dataset has no chunks to fetch, rather than tracebacking
FrancescAlted Aug 18, 2026
4fe79af
The api/chunk gap is fixed; note what a container leaf still lacks
FrancescAlted Aug 18, 2026
944ca80
Open a frame in two requests instead of four
FrancescAlted Aug 18, 2026
b2918c8
Notice a remote array that was replaced under a proxy's cache
FrancescAlted Aug 18, 2026
7b9aa62
Record the whole of the block-granularity work in the plan
FrancescAlted Aug 18, 2026
b6b7303
Plan byte ranges for container leaves
FrancescAlted Aug 18, 2026
e0cc7da
Say where a leaf's frame lies inside a .b2z
FrancescAlted Aug 18, 2026
460b060
Pin that a container leaf reads its blocks like any other array
FrancescAlted Aug 18, 2026
830f370
Record what container-leaf ranges landed as
FrancescAlted Aug 18, 2026
acf2d24
Cross off container-leaf ranges in the block-granularity plan
FrancescAlted Aug 18, 2026
438d2fc
Shim to reproduce itertools.batched for python 3.11
FrancescAlted Aug 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion RELEASE_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,47 @@ XXX version-specific blurb XXX
with the traffic, since a fetch in flight is now a block rather than a chunk.
`bench/ndarray/fsspec-block-granularity.py` measures both on any array.

* A `Proxy` over a `C2Array` reads **blocks** too, straight out of the stored
frame over HTTP byte ranges. Caterva2 serves a stored dataset from a file, so
the `Range` header is honoured and composes with the auth cookie; no new
endpoint is involved. On cat2.cloud's `kevlar-tomo.b2nd` a corner slice costs
0.031 MB instead of 2.723 MB, and a slice touching ten chunks takes 0.14 s
against 1.01 s. Three things add up to it: one pooled HTTP client instead of a
connection per request (0.162 s → 0.046 s each), fetches that overlap by
default as `afetch()` already did, and one request carrying the whole wave of
ranges (`multipart/byteranges`, which no object store offers). A dataset the
subscriber *computes* — a lazy expression, an HDF5 leaf, a `.b2z` member — is
fetched a whole chunk at a time as before; which it is costs at most one small
request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is
the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it
with a `read_range(offset, size)` to give any transport the same treatment.
Opening a remote frame through either of them now costs two requests instead
of four (0.237 s → 0.138 s against cat2.cloud), and one for a frame small
enough to arrive whole in the first read: the two reads that only measured the
next one are guessed at generously instead, since over a network a few hundred
bytes and a few kilobytes cost the same.
`bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset,
against a real subscriber or a stand-in it starts itself.

* `DictStore.member_window(key)` says where a leaf's frame lies inside a `.b2z`,
as `(offset, nbytes)`. A zip store keeps each external leaf uncompressed, so
those bytes are the frame that leaf would have been written as on its own --
which lets a reader take the window instead of the leaf: Caterva2 now serves a
container leaf from it, so a `Proxy` over `@public/tree.b2z/leaf` reads blocks
exactly as it does over a `.b2nd` (2.8x on a point read, 20x fewer bytes).
None where there is no window: a directory-backed store, an embedded leaf, a
`C2Array` reference.

* `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an
earlier run instead of failing on the existing file, so a proxy's cache can
outlive the process. The cache must come from a proxy over a source of the same
shape and dtype; anything else at that path raises. A cache that holds only
some blocks of a chunk keeps them across runs too.
some blocks of a chunk keeps them across runs too. Sources that can name the
bytes they read are held to that as well, so a remote array *replaced* while
keeping its shape is noticed rather than served stale: `FsspecNDSource` uses
fsspec's token and `C2Array` the subscriber's mtime, both free with metadata
they already fetch. Caches from earlier 4.11.1 development builds are not
adopted (the stamp moved to a `proxy-stamp` entry); pass `mode="w"` once.

* Querying a `utf8()` column through its FULL index no longer materializes the
index vocabulary. The query literal is turned into an alphabetical rank by
Expand Down
Loading
Loading