From c15bda7759496b74cb3754be9531d103de4fd376 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:22:35 +0200 Subject: [PATCH 01/25] Add the plan for block-granular Caterva2 reads Analysis of what it would take to give C2Array the block-granular reads blosc2.open(url, lazy=True) got in #701, and where the work goes. Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 314 ++++++++++++++++++++++++++++++++ 1 file changed, 314 insertions(+) create mode 100644 plans/cat2-block-granularity.md diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md new file mode 100644 index 000000000..4e6addb51 --- /dev/null +++ b/plans/cat2-block-granularity.md @@ -0,0 +1,314 @@ +# Block-Granular Reads For Caterva2 And `C2Array` + +Analysis and plan — nothing implemented. Written 2026-08-17, after +[plans/fsspec-blocks.md](fsspec-blocks.md) landed block fetching for fsspec URLs +(merged as PR #701). + +## The question + +`blosc2.open(url, lazy=True)` now fetches only the blocks a slice touches, and +gets 5-17x against an object store for the shapes that suit it. `C2Array`, the +other remote path, still fetches whole chunks: it reads them through Caterva2's +`api/chunk` endpoint, which is indexed by chunk number and has no notion of a +byte range. What would it take to give it the same thing, and where should the +work go — into blosc2, or into the Caterva2 server? + +**Verdict: into the client, mostly.** Caterva2 already serves byte ranges +properly for stored datasets, including authenticated ones, so no new endpoint +is needed. What is missing is a `read_range` on `C2Array`, a way to tell which +datasets support it, and a connection pool — plus two small server changes that +make the arrangement honest rather than lucky. + +## What was verified + +Against `https://cat2.cloud/demo` and against a local server run from +`~/ironArray/caterva2` (`CATERVA2_SECRET=c2sikrit cat2-server`, serving +`_caterva2/state/public/`). + +### Caterva2 serves real byte ranges, for free + +`fetch_data` (`caterva2/services/server.py:581`) returns `FileResponse` when the +whole of a stored dataset is asked for: + +```python +if ( + whole + and not isinstance( + array, blosc2.LazyArray | hdf5.HDF5Proxy | blosc2.NDField | blosc2.CTable + ) + and not filter + and inner_key is None +): + return FileResponse( + abspath, filename=abspath.name, media_type="application/octet-stream" + ) +``` + +Starlette's `FileResponse` implements RFC 7233 on its own: `Accept-Ranges: +bytes`, 206 with `Content-Range`, and `_handle_single_range` does +`await file.seek(start)` and reads only `end - start` bytes. So Caterva2 +inherits ranges from that one `return`. Measured on `kevlar-tomo.b2nd` +(14.44 MB, 10 chunks of ~1.44 MB, 47 blocks each), over the network to the demo: + +| request | median | bytes | +|---|---|---| +| full fetch | 1.654 s | 14.44 MB | +| **range 32 KB** | **0.085 s** | 0.03 MB | +| range 1.4 MB | 0.228 s | 1.44 MB | +| `api/chunk`, one chunk | 0.234 s | 1.37 MB | + +A small range costs 5% of the full fetch, and a large one costs what the same +bytes cost through `api/chunk`. Nothing is materialized per request. + +**Multipart works too**: `Range: bytes=0-31, 1000-1063, 5000-5099` returns 206 +`multipart/byteranges` (566 bytes for the three spans). No object store offers +that, and `block_plan` already computes exactly the coalesced range list it +wants — see phase 4. + +**Authentication composes.** On the local server, a `@personal` dataset uploaded +by a logged-in user answers a ranged request with 206 when the cookie is sent and +401 without it. The `FileResponse` branch runs after `split_and_resolve(path, +user)`, so ranges and auth are orthogonal. This is the fact that decides the +design: fsspec's HTTP filesystem cannot carry that cookie, and `C2Array` already +holds it. + +### The branch is narrow, and missing it is silent + +Everything else returns `StreamingResponse`, which has no range support at all. +Measured, on both servers: + +``` +GET api/fetch/@public/kevlar-tomo.b2nd?slice_=0:1 Range: bytes=0-31 + -> 200, 1,364,328 bytes, no Content-Range, no Accept-Ranges + +GET api/fetch/@personal/doubled.b2nd (a lazy expression) Range: bytes=0-0 + -> 200, 2,423,617 bytes +``` + +So lazy expressions, HDF5-backed datasets, `.b2z`/`.h5` members, `field=` and +`filter=` queries, and peer/provider datasets answer a one-byte request with the +whole body. `api/download` is the same. And clients do not defend themselves: +fsspec's HTTP filesystem, asked for 32 bytes of the first case, **returned +1,364,328 bytes in 2.7 s** and reported no error. A byte-range reader pointed at +such a dataset degrades into N full downloads while appearing to work. + +### `api/info` already says which is which + +A stored dataset reports `chunks`, `blocks` and `schunk`; a computed one reports +`expression` and `operands` and none of those: + +``` +@personal/personal.b2nd keys=['blocks', 'chunks', 'dtype', 'mtime', 'schunk', 'shape'] +@personal/doubled.b2nd keys=['dtype', 'expression', 'mtime', 'operands', 'shape'] +``` + +`C2Array` fetches `api/info` at construction already, so the discriminator costs +nothing. It is necessary but **not sufficient**: an HDF5 leaf or a `.b2z` member +may well report a geometry and still be served by `StreamingResponse`. The +authority has to be the status code of the first range read — see phase 2. + +### The prototype + +A ~40-line adapter, with no changes to blosc2 at all, reading a local +authenticated dataset block by block: + +``` +opened over Range+auth: (600, 600) float64 chunks=(300, 600) blocks=(30, 600) blocks/chunk=10 + server honours ranges (probe): True + opening cost: 4 requests, 253 bytes +slice [10:12, 30:40] via blocks: 2 requests, 121,214 bytes correct=True +the same data as C2Array does it: 1 request, 1,211,720 bytes +``` + +Ten blocks per chunk, ten times fewer bytes. End-to-end over the network, on the +demo's `kevlar-tomo.b2nd` with blosc2's shipped 1 MB threshold and no patching: +**1.364 MB in 1.477 s (chunks) against 0.015 MB in 0.269 s (blocks)**. + +It worked because `FsspecNDSource` never uses fsspec directly past construction: +it asks its filesystem object for `isdir`, `ukey` and `cat_file(path, start=, +end=)`, which is a three-method interface anything can implement. The prototype +supplied one over `httpx` and monkeypatched `blosc2.core._import_fsspec` to hand +it over. That monkeypatch is the only part that needs a supported replacement — +see phase 3. + +### `C2Array` builds a new HTTP client per request + +`_xget` calls `_httpx().get(...)`, and `_httpx()` returns the *module*, so every +request opens a connection and negotiates TLS. On the same chunk over the +network: + +``` +C2Array.get_chunk(0) 0.886 s (1.35, 0.89, 0.81) +same URL, pooled client 0.513 s (1.20, 0.51, 0.42) +``` + +~0.37 s per request of pure setup. Block mode issues *more, smaller* requests, +which is exactly what per-request handshakes punish. `aget_chunk` already keeps a +reused `AsyncClient`; the sync path never got the same treatment. + +## What not to build + +**A block endpoint** (`api/block/{path}?nchunk=&nblock=`). It duplicates what +Range already does for stored data, and for computed data it solves the wrong +problem: a block only exists as bytes when the array is stored in blocks, so for +a lazy expression or an HDF5 proxy the server must compute a whole chunk before +it can hand one over — network saved, server work unchanged. Caterva2 already has +the better primitive there, and it is finer than blocks: `slice_` asks for +exactly the region wanted and computes only that. `C2Array.__getitem__` uses it +already. + +The rule that falls out, and that the plan follows throughout: + +> **Blocks where the bytes already exist in blocks; `slice_` where they must be +> computed.** The split is exactly the `FileResponse` / `StreamingResponse` line +> the server already has. + +## Plan + +### Phase 1 — Pool the HTTP client in `C2Array` (caterva2) + +Smallest change, largest certain payoff, and independent of everything else. +`_xget`/`_xpost` should share a module-level `httpx.Client` (thread-safe, with +`limits=` set) instead of calling `httpx.get`. Measured: ~0.37 s per request +saved over a WAN link, on the *existing* chunk path. + +Care: the client must not capture auth headers globally (they are per call), and +a long-lived client needs `timeout` and connection limits configured. Closing it +at interpreter exit is nice but not required. + +**Do this first even if the rest is never built.** + +### Phase 2 — A capability check on the client (caterva2) + +`C2Array` gains a private `_ranges_ok` state with three values: unknown, yes, no. + +- From `api/info`, already fetched at construction: no `chunks`/`blocks`/`schunk` + (i.e. an `expression` dataset) means **no** without any request. +- Otherwise unknown, resolved by the first range read: 206 with a `Content-Range` + means **yes**; a 200 means **no**, and the source falls back to whole chunks for + the life of the object. + +The fallback must be permanent and silent-safe: never retry ranges on a dataset +that answered 200, or every read pays a full download to rediscover it. Note the +probe is expensive exactly where it fails (a 200 carries the whole body — 2.4 MB +for the lazy dataset above), which is why phase 5 matters. + +### Phase 3 — A supported seam in blosc2, then `C2Array` block support + +The prototype showed the machinery is already generic; what it lacks is a +sanctioned way in. Two options, in increasing order of tidiness: + +- **(a) A `fs=` argument on `FsspecNDSource`.** Two lines: when given, skip + `url_to_fs` and use the object as is. Anything with `isdir`, `ukey` and + `cat_file(path, start=, end=)` then works, `C2Array` included. Minimal, and + slightly dishonest — the class is named for fsspec. +- **(b) Split the class.** Lift the frame parsing and block planning + (`_read_frame_index`, `_chunk_extents`, `_block_extents`, `chunk_layout`, + `block_plan`, `wants_blocks`, `get_chunk`, `_special_chunk`) into a + `ByteRangeNDSource` base whose only abstract method is `read_range(offset, + size)`. `FsspecNDSource` becomes that base plus an fsspec transport; + `C2NDSource` becomes the same base plus an `httpx` transport that carries the + Caterva2 cookie. **Recommended**: there are now two implementations, which is + when a base class earns its place, and it puts the format knowledge in one + place where it has already needed four corrections (dictionaries, + variable-length blocks, non-extended headers, repeated-value chunks). + +On the Caterva2 side, `C2Array` then either implements the base itself or exposes +`read_range` and lets `blosc2.Proxy` do the rest. The URL is +`{urlbase}/api/fetch/{path}` with the auth cookie; `blocks_per_chunk` comes from +`self.blocks`, which `C2Array` already has. + +Note what does *not* change: the thresholds, the coalescing, the splicing into a +partially-filled cache chunk, the persistent cache and its bitmap, `max_concurrency` +— all of it is in `Proxy` and the base, and all of it is already tested. + +### Phase 4 — Multipart ranges (blosc2, optional) + +`block_plan` returns a coalesced list of ranges per chunk. Caterva2 answers +`multipart/byteranges`, so those could go out as **one request** instead of one +per run — a strictly better deal than any object store offers, and the thing that +would make Caterva2 the best backend for block reads rather than merely an equal +one. Needs a multipart parser on the client (~40 lines) and only pays where a +slice touches several disjoint runs of a chunk. Measure before building: with the +two-wave design a chunk's runs already go out in parallel, so this converts +parallel requests into one, which matters for per-request cost, not latency. + +Do not build this unless phase 5 is done: batching into a request shape the +server may stop honouring is a bad trade. + +### Phase 5 — Make the streaming paths honest (caterva2) + +Small, and it protects every future client: + +- set `Accept-Ranges: none` on the `StreamingResponse` returns in `fetch_data` + and `download_data`; +- answer a `Range` header on those paths with **416** (or 400) rather than + ignoring it. + +Then a client learns the answer in one cheap exchange instead of a full download, +phase 2's probe becomes nearly free, and the silent N-full-downloads failure mode +stops existing. If the range support is to be a documented feature rather than an +accident, this is the change that makes it one. + +## Risks and open questions + +- **The `FileResponse` branch is load-bearing but incidental.** Nothing in + Caterva2's tests asserts that a stored dataset is served by `FileResponse`, so + a refactor could turn it into a `StreamingResponse` and silently halve every + block client's performance. If phase 3 ships, Caterva2 wants a test that asserts + 206 and `Content-Range` on a stored `.b2nd`. +- **HDF5 leaves and `.b2z` members are unverified.** The code puts them on the + streaming path (`inner_key is not None`), but the local server returned 404 for + the HDF5 leaf paths I tried, so I could not confirm what they report in + `api/info`. If either reports `chunks`/`blocks` while being streamed, the + info-based discriminator alone would be wrong — which is why the 206 check is + the authority. +- **Peer/provider datasets** go through `provider.fetch` and a + `StreamingResponse`; same treatment, worth a look when peers matter. +- **Quota and accounting.** Many small ranged GETs replace one chunk GET. If + Caterva2 ever meters per request rather than per byte, block clients look + expensive while transferring far less. Worth deciding before it surprises + someone. +- **The gain depends on chunk size, as it does for S3.** blosc2 declines to take + a chunk apart below 1 MB compressed, so small-chunked datasets keep today's + behaviour exactly. `cube-1k-1k-1k.b2nd` (~102 KB per chunk) would never use + blocks; `kevlar-tomo.b2nd` (1.44 MB) always would. + +## Rough sizing + +| | where | size | +|---|---|---| +| 1. pooled client | caterva2 | ~10 lines | +| 2. capability check | caterva2 | ~30 lines + tests | +| 3a. `fs=` seam | blosc2 | ~5 lines | +| 3b. `ByteRangeNDSource` split | blosc2 | ~80 lines moved, ~20 new | +| 3. `C2Array` block support | caterva2 | ~60 lines + tests | +| 4. multipart | blosc2 | ~60 lines, only if measured | +| 5. honest streaming paths | caterva2 | ~10 lines + a test | + +## Reproducing the measurements + +A local server, which is what phases 2 and 5 need to be tested against: + +```sh +cd ~/ironArray/caterva2 +CATERVA2_SECRET=c2sikrit cat2-server & # serves _caterva2/state/public/ +CATERVA2_SECRET=c2sikrit cat2-admin adduser probe@example.com foobar11 +``` + +Then, for the range behaviour of any dataset: + +```python +import httpx + +r = httpx.get( + "http://localhost:8000/api/fetch/@public/kevlar-tomo.b2nd", + headers={"Range": "bytes=0-31"}, +) +print(r.status_code, r.headers.get("content-range"), len(r.content)) +# 206 'bytes 0-31/14435027' 32 -> served by FileResponse, blocks will work +# 200 None 14435027 -> served by a body builder, blocks must not be used +``` + +`bench/ndarray/fsspec-block-granularity.py` measures the touch ratios of any +local array, which is what decides whether a given dataset would benefit at all. From 7bb0eff1782b54a9788d1314cf9674ff4278f815 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:22:42 +0200 Subject: [PATCH 02/25] Pool the HTTP client C2Array requests go through httpx.get() builds a client, opens a connection and negotiates TLS per call. Against cat2.cloud that is 0.162 s per request against 0.046 s through a pooled client, on the existing chunk path -- 3.5x, and it grows in weight as reads get smaller, which block-granular reads will make them. The client is shared and thread-safe, which is what lets Proxy fan its fetches out, but it never keeps a cookie: auth belongs to the C2Array being read, and arrays with different tokens (or none) share the client. Login keeps a client of its own for the same reason. Co-Authored-By: Claude Opus 5 --- src/blosc2/c2array.py | 66 ++++++++++++++++++++++++++++- tests/ndarray/test_c2array_async.py | 6 ++- 2 files changed, 69 insertions(+), 3 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index ba85c2897..8d9874b86 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -7,7 +7,9 @@ from __future__ import annotations +import atexit import os +import threading from contextlib import contextmanager from typing import TYPE_CHECKING @@ -38,6 +40,64 @@ def _httpx(): return httpx +_client = None +_client_lock = threading.Lock() + + +def _forgetful_cookies(httpx): + """A cookie jar that never keeps anything, for the shared client. + + A client of its own per request could not carry a cookie from one request to + the next; a shared one can, and must not: the token belongs to the C2Array + being read, and arrays with different tokens (or none) share this client. A + `Set-Cookie` from any response would otherwise start authorizing requests + that asked for none. + """ + + class _NoCookies(httpx.Cookies): + def extract_cookies(self, response): + pass + + return _NoCookies() + + +def _sync_client(): + """The process-wide HTTP client every synchronous request goes through. + + `httpx.get()` builds a client, opens a connection and negotiates TLS for + each call and throws all of it away afterwards, which over a WAN link + measured ~0.37 s per request -- most of what a small read costs, and paid + once per chunk. A pooled client keeps the connection alive between + requests; it is thread-safe, which is what lets `Proxy` fan its fetches out. + + Auth stays per call rather than on the client: the cookie belongs to the + C2Array being read, and several of them (with different tokens, or none) + share this one client. + """ + global _client + if _client is None: + with _client_lock: + if _client is None: + httpx = _httpx() + # More connections than `Proxy`'s default concurrency, so that a + # caller raising it does not queue on the pool; keepalive covers + # the fan-out of one fetch, which is what there is to reuse + _client = httpx.Client( + timeout=TIMEOUT, + limits=httpx.Limits(max_connections=64, max_keepalive_connections=32), + cookies=_forgetful_cookies(httpx), + ) + return _client + + +@atexit.register +def _close_sync_client(): + global _client + if _client is not None: + _client.close() + _client = None + + @contextmanager def c2context( *, @@ -121,7 +181,7 @@ def _auth_headers(auth_token, headers=None): def _xget(url, params=None, headers=None, auth_token=None, timeout=TIMEOUT): headers = _auth_headers(auth_token, headers) - response = _httpx().get(url, params=params, headers=headers, timeout=timeout) + response = _sync_client().get(url, params=params, headers=headers, timeout=timeout) response.raise_for_status() return response @@ -129,7 +189,7 @@ def _xget(url, params=None, headers=None, auth_token=None, timeout=TIMEOUT): def _xpost(url, json=None, auth_token=None, timeout=TIMEOUT): auth_token = auth_token or _subscriber_data["auth_token"] headers = {"Cookie": auth_token} if auth_token else None - response = _httpx().post(url, json=json, headers=headers, timeout=timeout) + response = _sync_client().post(url, json=json, headers=headers, timeout=timeout) response.raise_for_status() return response.json() @@ -144,6 +204,8 @@ def _sub_url(urlbase, path): def login(username, password, urlbase): url = _sub_url(urlbase, "auth/jwt/login") creds = {"username": username, "password": password} + # Not the pooled client: this is the one request whose Set-Cookie matters, + # and it belongs to the caller rather than to every later request resp = _httpx().post(url, data=creds, timeout=TIMEOUT) resp.raise_for_status() return "=".join(list(resp.cookies.items())[0]) diff --git a/tests/ndarray/test_c2array_async.py b/tests/ndarray/test_c2array_async.py index af4978520..4556eaa5f 100644 --- a/tests/ndarray/test_c2array_async.py +++ b/tests/ndarray/test_c2array_async.py @@ -27,6 +27,8 @@ def json(self): class _FakeHttpx: + """Stands in for both the httpx module and the pooled sync client.""" + HTTPStatusError = Exception def __init__(self, meta): @@ -47,7 +49,9 @@ def fake_c2array(monkeypatch): "dtype": str(array.dtype), "schunk": {"cparams": {"typesize": array.dtype.itemsize}}, } - monkeypatch.setattr(c2array_mod, "_httpx", lambda: _FakeHttpx(meta)) + fake = _FakeHttpx(meta) + monkeypatch.setattr(c2array_mod, "_httpx", lambda: fake) + monkeypatch.setattr(c2array_mod, "_sync_client", lambda: fake) c2 = blosc2.C2Array("@public/fake.b2nd", urlbase="http://fake-server/") c2._chunks_source = array # stash the real array to serve chunk bytes from return c2 From 422ba4b49d513754d66d7d0696a6d567b1a914fe Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:26:33 +0200 Subject: [PATCH 03/25] Lift the frame reading out of FsspecNDSource into ByteRangeNDSource Everything FsspecNDSource knows about taking a Blosc2 frame apart -- the header, the chunk offsets, which chunks are worth reading block by block, where those blocks are and how few requests they fit in -- is the frame format and nothing else. It now lives in a base class whose only abstract method is read_range(offset, size); FsspecNDSource is that base plus four lines of fsspec, and C2Array will be the same base plus HTTP ranges. The frame index is read through read_range too, so a subclass has one seam to implement rather than two. That is what the test churn is: reads made while opening, and the read behind a whole-chunk fetch, now show up in the traffic the tests count. Co-Authored-By: Claude Opus 5 --- doc/reference/byterangendsource.rst | 19 +++++ doc/reference/classes.rst | 1 + doc/reference/fsspecndsource.rst | 9 +-- src/blosc2/__init__.py | 2 + src/blosc2/proxy.py | 105 +++++++++++++++++++--------- tests/test_fsspec.py | 29 +++++--- 6 files changed, 117 insertions(+), 48 deletions(-) create mode 100644 doc/reference/byterangendsource.rst diff --git a/doc/reference/byterangendsource.rst b/doc/reference/byterangendsource.rst new file mode 100644 index 000000000..af57c43f2 --- /dev/null +++ b/doc/reference/byterangendsource.rst @@ -0,0 +1,19 @@ +.. _ByteRangeNDSource: + +ByteRangeNDSource +================= + +A :ref:`ProxyNDSource` that serves the chunks -- and the single blocks -- of a +Blosc2 frame it can read byte ranges of, instead of transferring the whole +container. It knows the frame format and nothing about where the frame lives: +subclasses supply ``read_range(offset, size)`` and nothing else. +:ref:`FsspecNDSource` reads through fsspec, and :ref:`C2Array` reads over HTTP +ranges from a Caterva2 subscriber. For other sources, see :ref:`ProxyNDSource` +and :ref:`ProxySource`. + +.. currentmodule:: blosc2 + +.. autoclass:: ByteRangeNDSource + :members: + :exclude-members: all, any, max, mean, min, prod, std, sum, var + :member-order: groupwise diff --git a/doc/reference/classes.rst b/doc/reference/classes.rst index 1a33777fd..4efd1a25a 100644 --- a/doc/reference/classes.rst +++ b/doc/reference/classes.rst @@ -139,6 +139,7 @@ container APIs above. proxy proxysource proxyndsource + byterangendsource fsspecndsource simpleproxy embed_store diff --git a/doc/reference/fsspecndsource.rst b/doc/reference/fsspecndsource.rst index 399e0eb83..deb2a6fb6 100644 --- a/doc/reference/fsspecndsource.rst +++ b/doc/reference/fsspecndsource.rst @@ -3,10 +3,11 @@ FsspecNDSource ============== -A :ref:`ProxyNDSource` that serves the chunks of a Blosc2 frame living behind an -fsspec URL, reading each one with a range request instead of transferring the -whole container. For other sources, see :ref:`ProxyNDSource` and -:ref:`ProxySource`. +A :ref:`ByteRangeNDSource` that serves the chunks of a Blosc2 frame living +behind an fsspec URL, reading each one with a range request instead of +transferring the whole container. Everything about the frame format, block +granularity included, lives in the base class; this adds the fsspec transport. +For other sources, see :ref:`ProxyNDSource` and :ref:`ProxySource`. ``examples/ndarray/rw-fsspec.py`` is a runnable walkthrough of this and the other two ways to read an fsspec URL, and of writing one back. diff --git a/src/blosc2/__init__.py b/src/blosc2/__init__.py index 648014b58..835b1ac0d 100644 --- a/src/blosc2/__init__.py +++ b/src/blosc2/__init__.py @@ -593,6 +593,7 @@ def _raise(exc): ProxySource, ProxyNDSource, ProxyNDField, + ByteRangeNDSource, FsspecNDSource, SimpleProxy, jit, @@ -879,6 +880,7 @@ def _raise(exc): "NDArray", "NDField", "Operand", + "ByteRangeNDSource", "FsspecNDSource", "Proxy", "ProxyNDField", diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index feaa35491..3448fcb3e 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -1116,14 +1116,14 @@ def _read_frame_index(f) -> tuple[bytes, list, np.ndarray]: class _RangeReader: """The seek/read pair `_read_frame_index` needs, served by exact range requests.""" - def __init__(self, fs, path: str): - self._fs, self._path, self._pos = fs, path, 0 + def __init__(self, read_range): + self._read_range, self._pos = read_range, 0 def seek(self, pos: int) -> None: self._pos = pos def read(self, size: int) -> bytes: - data = self._fs.cat_file(self._path, start=self._pos, end=self._pos + size) + data = self._read_range(self._pos, size) self._pos += len(data) return data @@ -1152,17 +1152,11 @@ def _chunk_extents(offsets: np.ndarray, header: list) -> np.ndarray: return np.minimum(extents, header[8] + blosc2.MAX_OVERHEAD) -class FsspecNDSource(ProxyNDSource): +class ByteRangeNDSource(ProxyNDSource): """A :ref:`Proxy` source that serves parts of a remote Blosc2 frame. The frame stays where it is: only its header, its chunk offsets, and what a - slice actually touches ever cross the network. This is what - ``blosc2.open(url, lazy=True)`` builds; wrap it in a :ref:`Proxy` by hand - when the cache belongs at a path of your choosing rather than inside - ``cache_storage``:: - - src = blosc2.FsspecNDSource("s3://bucket/big.b2nd") - a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a") + slice actually touches ever cross the network. A chunk large enough to be worth taking apart is read block by block -- :meth:`chunk_layout` fetches the offsets of its blocks, :meth:`block_plan` @@ -1172,14 +1166,25 @@ class FsspecNDSource(ProxyNDSource): to save there; :meth:`wants_blocks` decides which is which without reading anything. + Everything above is the Blosc2 frame format and nothing else, so a subclass + only has to say how to read bytes: :meth:`read_range` is the one abstract + method, and the transport behind it decides nothing about the rest. + :ref:`FsspecNDSource` reads them with fsspec, and :ref:`C2Array` reads them + over HTTP ranges from a Caterva2 subscriber, carrying its auth cookie. + + A subclass sets its transport up first and then calls this constructor, + which reads the frame's header and chunk offsets through it (three small + reads). It may also set a ``stamp``, anything that names the exact bytes it + reads, so that :ref:`Proxy` can tell a cache built from other bytes. + Contiguous frames carrying a ``b2nd`` metalayer only, which is what :func:`blosc2.asarray` and friends write to a single file. Sparse frames and - ``.b2d`` stores are directories; open those with ``cache_storage=``. + ``.b2d`` stores are directories, and cannot be read this way. Parameters ---------- urlpath: str - The fsspec URL of the frame. + Where the frame is, for error messages and for the caller to read back. max_concurrency: int, optional How many fetches the enclosing :ref:`Proxy` may run at once. Every chunk or block costs one range request, so against an object store a slice is @@ -1189,28 +1194,15 @@ class FsspecNDSource(ProxyNDSource): the thread pool costs about 10 microseconds per chunk and saves nothing. """ - def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): - from blosc2.core import _import_fsspec + stamp = None + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self.max_concurrency = max_concurrency - fsspec = _import_fsspec(urlpath) - fs, path = fsspec.url_to_fs(urlpath) - if fs.isdir(path): - raise NotImplementedError( - f"{urlpath} is a directory (a sparse frame or a store), which cannot be read " - "chunk by chunk; open it with cache_storage= instead" - ) self.urlpath = urlpath - self._fs, self._path = fs, path - # Identifies the remote bytes, so a cache built against them can tell it - # has gone stale -- and chunk offsets from a replaced frame are garbage. - # fsspec's own token, rather than a tuple of the metadata fields we guess - # a backend exposes: memory:// has no mtime, which left it size-only. - self.stamp = fs.ukey(path) - # Exact ranges, not fs.open(): a buffered handle reads a whole block per + # Exact ranges, not a file handle: a buffered one reads a whole block per # seek (50 MiB on s3fs by default), which would undo the point of a lazy # open. Chunk reads are stateless, so nothing here is shared between threads - raw, header, self._offsets = _read_frame_index(_RangeReader(fs, path)) + raw, header, self._offsets = _read_frame_index(_RangeReader(self.read_range)) self._chunksize = header[8] self._extents = _chunk_extents(self._offsets, header) try: @@ -1264,12 +1256,19 @@ def get_chunk(self, nchunk: int) -> bytes: offset = int(self._offsets[nchunk]) if offset < 0: return self._special_chunk(offset) - data = self._fs.cat_file(self._path, start=offset, end=offset + int(self._extents[nchunk])) + data = self.read_range(offset, int(self._extents[nchunk])) return data[: struct.unpack(" bytes: - """The bytes at [*offset*, *offset* + *size*) of the frame.""" - return self._fs.cat_file(self._path, start=offset, end=offset + size) + """The bytes at [*offset*, *offset* + *size*) of the frame. + + The whole of the transport: everything else here is the frame format. + Fewer bytes may come back only at the end of the frame; anything else is + an error, since the caller has no way to ask for the rest. Must be safe + to call from several threads at once, which is what lets :ref:`Proxy` + overlap the fetches of one slice. + """ def wants_blocks(self, nchunk: int, nwanted: int) -> bool: """Whether fetching *nwanted* blocks of a chunk beats fetching all of it. @@ -1398,6 +1397,46 @@ def _special_chunk(self, offset: int) -> bytes: ) +class FsspecNDSource(ByteRangeNDSource): + """A :ref:`ByteRangeNDSource` reading its frame through fsspec. + + This is what ``blosc2.open(url, lazy=True)`` builds; wrap it in a + :ref:`Proxy` by hand when the cache belongs at a path of your choosing + rather than inside ``cache_storage``:: + + src = blosc2.FsspecNDSource("s3://bucket/big.b2nd") + a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a") + + Parameters + ---------- + urlpath: str + The fsspec URL of the frame. + max_concurrency: int, optional + As in :ref:`ByteRangeNDSource`. + """ + + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): + from blosc2.core import _import_fsspec + + fsspec = _import_fsspec(urlpath) + fs, path = fsspec.url_to_fs(urlpath) + if fs.isdir(path): + raise NotImplementedError( + f"{urlpath} is a directory (a sparse frame or a store), which cannot be read " + "chunk by chunk; open it with cache_storage= instead" + ) + self._fs, self._path = fs, path + # Identifies the remote bytes, so a cache built against them can tell it + # has gone stale -- and chunk offsets from a replaced frame are garbage. + # fsspec's own token, rather than a tuple of the metadata fields we guess + # a backend exposes: memory:// has no mtime, which left it size-only. + self.stamp = fs.ukey(path) + super().__init__(urlpath, max_concurrency) + + def read_range(self, offset: int, size: int) -> bytes: + return self._fs.cat_file(self._path, start=offset, end=offset + size) + + class ProxyNDField(blosc2.Operand): def __init__(self, proxy: Proxy, field: str): self.proxy = proxy diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index c7103f97c..b988f0dcb 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -733,7 +733,13 @@ def any_chunk_wants_blocks(monkeypatch): def _traffic(monkeypatch): - """Record every range read and whole-chunk read a source makes.""" + """Record every range read and whole-chunk read a source makes. + + Everything the source reads goes through `read_range`, opening the frame and + fetching a whole chunk included, so `reads` counts requests and a + whole-chunk fetch appears in both lists. Install it after the open where + the frame index would otherwise be counted in. + """ reads, chunks = [], [] for name, log in (("read_range", reads), ("get_chunk", chunks)): orig = getattr(blosc2.FsspecNDSource, name) @@ -758,9 +764,9 @@ def test_lazy_fetches_only_touched_blocks(monkeypatch): data, a = _incompressible((600, 600), (300, 600), (30, 600)) cbytes = a.schunk.cbytes // a.schunk.nchunks assert cbytes > blosc2.proxy.BLOCK_MIN_CBYTES - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("blocks.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) # after the open, which reads the frame index assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) # One read for the block offsets, one for the single block the slice lands in assert len(reads) == 2 @@ -772,23 +778,23 @@ def test_lazy_small_chunks_are_fetched_whole(monkeypatch): # Below the threshold a chunk is one cheap request, so blocks would only add # a round trip; nothing must go looking for block offsets a = blosc2.arange(0, 1000, dtype="i4", chunks=(100,)) - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("smallblocks.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[150:250], a[150:250]) - assert not reads assert len(chunks) == 2 + assert len(reads) == len(chunks) # one request each, none of them for block offsets def test_lazy_whole_array_skips_the_block_path(monkeypatch, any_chunk_wants_blocks): # Wanting every block of a chunk is what fetching the chunk already does data, a = _incompressible((200, 200), (100, 200), (10, 200)) - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("wholeblocks.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[:], data) - assert not reads assert len(chunks) == 2 + assert len(reads) == len(chunks) # one request each, none of them for block offsets @pytest.mark.parametrize( @@ -835,16 +841,16 @@ def test_lazy_block_cache_survives_reopen(tmp_path, monkeypatch, any_chunk_wants p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - fetched = len(reads) del p # A partly filled chunk survives, so the blocks in it do not travel again p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + reopened = len(reads) # every open reads the frame index afresh assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - assert len(reads) == fetched + assert len(reads) == reopened # ... and the ones missing from it still do assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) - assert len(reads) > fetched + assert len(reads) > reopened assert np.array_equal(p[...], data) @@ -863,12 +869,13 @@ def test_lazy_blocks_fall_back_for_memcpyed_chunks(monkeypatch, any_chunk_wants_ # A memcpyed chunk stores its blocks raw and has no offsets section to read data = np.random.default_rng(0).integers(0, 256, (300, 300), dtype="u1") a = blosc2.asarray(data, chunks=(150, 300), blocks=(15, 300), cparams={"clevel": 0}) - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("memcpyed.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) - assert len(reads) == 1 # the offsets are read, and say there is nothing to skip assert len(chunks) == 1 + # The offsets are read, say there is nothing to skip, and the chunk follows + assert len(reads) == 2 def test_lazy_blocks_with_run_length_chunks(monkeypatch, any_chunk_wants_blocks): From d541eb3a170c617d8287cfeff78165d6cfa97c43 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:34:11 +0200 Subject: [PATCH 04/25] Read a C2Array's blocks over HTTP ranges Caterva2 serves a stored dataset with a FileResponse, which implements RFC 7233 on its own: a ranged request is answered 206 with the bytes asked for, seeked to in the file, and the auth cookie composes with it. So no new endpoint is needed for a Proxy over a C2Array to fetch the blocks a slice touches rather than the chunks they live in. On cat2.cloud's kevlar-tomo.b2nd a corner slice costs 0.031 MB instead of 2.723 MB -- 88x fewer bytes, and the same wall time on a fast link, where a round trip and a megabyte cost about the same. C2NDSource is ByteRangeNDSource plus that transport; C2Array grows the five members Proxy looks for and delegates them to it, so every existing Proxy(C2Array(...)) gets this without asking. What it costs when it does not apply is the point of the two gates before it: api/info rules out a dataset the subscriber computes (it reports an expression where a stored one reports a geometry) and one whose chunks are too small to be worth taking apart, both without a request, so those datasets behave exactly as they did. Everything else is decided by the status code of the first range read, once: a streamed dataset answers 200 with the whole body, which read_range refuses without reading it off the socket, and the array keeps to api/chunk for good. Co-Authored-By: Claude Opus 5 --- doc/reference/c2array.rst | 18 ++ src/blosc2/__init__.py | 3 +- src/blosc2/c2array.py | 126 +++++++++++ tests/ndarray/test_c2array_blocks.py | 321 +++++++++++++++++++++++++++ 4 files changed, 467 insertions(+), 1 deletion(-) create mode 100644 tests/ndarray/test_c2array_blocks.py diff --git a/doc/reference/c2array.rst b/doc/reference/c2array.rst index cfa6e7cee..da165991a 100644 --- a/doc/reference/c2array.rst +++ b/doc/reference/c2array.rst @@ -5,6 +5,15 @@ C2Array This is a class for remote arrays. This kind of array can also work as operand on a LazyExpr, LazyUDF or reduction. +Wrapped in a :ref:`Proxy`, a stored remote array is read at block granularity: +the proxy asks for the blocks a slice touches rather than the chunks they live +in, which for a multi-megabyte chunk is a small fraction of the bytes. That +rests on the subscriber serving the dataset from a file, ``Range`` header and +auth cookie both honoured; a dataset it computes instead (a lazy expression, an +HDF5 leaf) is fetched a whole chunk at a time, as everything was before. Which +one this is takes at most one request to find out, and is decided once -- +:meth:`C2Array.block_source` is what answers it. + .. currentmodule:: blosc2 @@ -28,6 +37,15 @@ This is a class for remote arrays. This kind of array can also work as operand o .. automethod:: __getitem__ +.. _C2NDSource: + +C2NDSource class +---------------- +.. autoclass:: C2NDSource + :members: + :member-order: groupwise + + .. _URLPath: URLPath class diff --git a/src/blosc2/__init__.py b/src/blosc2/__init__.py index 835b1ac0d..d6e965463 100644 --- a/src/blosc2/__init__.py +++ b/src/blosc2/__init__.py @@ -572,7 +572,7 @@ def _raise(exc): from .ref import Ref from .b2objects import open_b2object -from .c2array import c2context, C2Array, URLPath +from .c2array import c2context, C2Array, C2NDSource, URLPath from .dsl_kernel import DSLSyntaxError, DSLKernel, dsl_kernel, validate_dsl, validate_dsl_jit from .lazyexpr import ( @@ -853,6 +853,7 @@ def _raise(exc): "group_reduce", # Classes "C2Array", + "C2NDSource", "Column", "CParams", "CTable", diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index 8d9874b86..fd7bdbf3c 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -8,6 +8,7 @@ from __future__ import annotations import atexit +import math import os import threading from contextlib import contextmanager @@ -22,6 +23,10 @@ from blosc2.b2objects import encode_b2object_payload, make_b2object_carrier, write_b2object_payload from blosc2.info import InfoReporter, format_nbytes_info +# blosc2/__init__ imports this module before blosc2.proxy, so this pulls proxy in +# early; it is safe because proxy only reaches into the package at call time +from blosc2.proxy import REMOTE_MAX_CONCURRENCY, ByteRangeNDSource + _subscriber_data = { "urlbase": os.environ.get("BLOSC_C2URLBASE"), "auth_token": "", @@ -255,6 +260,49 @@ def slice_to_string(slice_): return ", ".join(slice_parts) +_UNTRIED = object() +"""A block source that has not been asked for yet, as against one that failed.""" + + +class _NotRanged(Exception): + """The subscriber answered a range request with something other than a 206.""" + + +class C2NDSource(ByteRangeNDSource): + """The frame behind a :ref:`C2Array`, read over HTTP byte ranges. + + Caterva2 serves a *stored* dataset with a Starlette ``FileResponse``, which + implements RFC 7233 by itself: a ranged request comes back 206 with only the + bytes asked for, seeked to in the file rather than materialized, and the auth + cookie composes with it. That is everything :ref:`ByteRangeNDSource` needs, + so a slice costs the blocks it touches instead of the chunks they live in. + + A dataset the subscriber *builds* -- a lazy expression, an HDF5 leaf, a + ``.b2z`` member -- is streamed instead, and a streamed response ignores the + ``Range`` header and answers with the whole body. :meth:`read_range` refuses + such an answer without reading it off the socket, and :ref:`C2Array` then + keeps to whole chunks for good. Which is why this is built through + :meth:`C2Array.block_source` rather than directly: the fallback belongs with + the array, whose ``api/chunk`` path works for every dataset there is. + """ + + def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY): + self._url = _sub_url(array.urlbase, f"api/fetch/{array.path}") + self._auth_token = array.auth_token + super().__init__(self._url, max_concurrency) + + def read_range(self, offset: int, size: int) -> bytes: + headers = _auth_headers(self._auth_token, {"Range": f"bytes={offset}-{offset + size - 1}"}) + with _sync_client().stream("GET", self._url, headers=headers) as response: + if response.status_code != 206: + # Whatever this is, it is not the bytes that were asked for: a 200 + # carries the whole dataset, which is the download this exists to + # avoid, so leave the body unread on the socket + raise _NotRanged(f"{self._url} answered {response.status_code} to a Range request") + response.read() + return response.content + + class C2Array(blosc2.Operand): """Remote compressed NDArray accessed from a Caterva2 server.""" @@ -305,6 +353,10 @@ def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | N self.auth_token = auth_token self._aclient = None # lazy async client, shared across aget_chunk calls + # The block-reading source, built on first use: _UNTRIED, None (this + # dataset cannot be read in ranges) or a C2NDSource + self._block_source = _UNTRIED + self._block_lock = threading.Lock() # Try to 'open' the remote path try: @@ -504,6 +556,80 @@ async def aclose(self) -> None: await self._aclient.aclose() self._aclient = None + # -- Block-granular reads. A :ref:`Proxy` uses these to fetch the blocks a + # slice touches instead of whole chunks, wherever that is the cheaper way + # round; every one of them falls back to `get_chunk` when it is not. + + @property + def blocks_per_chunk(self) -> int: + """How many blocks a chunk of the remote array holds. + + Geometry, and `api/info` already carries it, so this costs no request: + chunks are padded to whole blocks, so every chunk holds the same number + of them, edge chunks included. + """ + blocks = self.blocks + if not all(blocks): # an empty array partitions into nothing + return 1 + return math.prod(math.ceil(c / b) for c, b in zip(self.chunks, blocks, strict=True)) + + def block_source(self) -> C2NDSource | None: + """The frame reader behind the block methods, or None if there is none. + + Built on the first request for it and never rebuilt. The fallback has to + be permanent: a subscriber that streams this dataset answers a range + request with the whole body, so retrying would pay a full download to + rediscover the same answer. + """ + if self._block_source is _UNTRIED: + with self._block_lock: + if self._block_source is _UNTRIED: + self._block_source = self._open_block_source() + return self._block_source + + def _open_block_source(self) -> C2NDSource | None: + """Decide, at whatever cost it takes, whether this dataset serves ranges.""" + # `api/info` rules out a dataset the subscriber computes for nothing: a + # stored one reports its geometry where a lazy expression reports + # `expression` and `operands` + if not all(key in self.meta for key in ("chunks", "blocks", "schunk")): + return None + # Nor is a frame of small chunks worth an index read: blosc2 declines to + # take a chunk below BLOCK_MIN_CBYTES apart, so nothing here would ever + # use a block, and the dataset keeps exactly the behaviour it had before + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) + if not nchunks or self.cbytes / nchunks < blosc2.proxy.BLOCK_MIN_CBYTES: + return None + # Whether a dataset that reports a geometry is *served* from a file is + # something only the answer to a range request can say: an HDF5 leaf or a + # `.b2z` member reports one and is streamed all the same + try: + return C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) + except (_NotRanged, ValueError, NotImplementedError, RuntimeError, _httpx().HTTPError): + # Not ranged, not a contiguous frame, not an NDArray, or not + # reachable: whole chunks work for all of those + return None + + def wants_blocks(self, nchunk: int, nwanted: int) -> bool: + """Whether fetching *nwanted* blocks of a chunk beats fetching all of it.""" + source = self.block_source() + return source is not None and source.wants_blocks(nchunk, nwanted) + + def chunk_layout(self, nchunk: int): + """Where the blocks of a chunk are; see :meth:`ByteRangeNDSource.chunk_layout`.""" + return self.block_source().chunk_layout(nchunk) + + def block_plan(self, nchunk: int, nblocks: Sequence[int]) -> list[tuple[int, int, tuple]]: + """The range reads covering *nblocks*; see :meth:`ByteRangeNDSource.block_plan`.""" + return self.block_source().block_plan(nchunk, nblocks) + + def read_range(self, offset: int, size: int) -> bytes: + """The bytes at [*offset*, *offset* + *size*) of the remote frame.""" + source = self.block_source() + if source is None: + raise ValueError(f"{self.path} is not served in byte ranges by {self.urlbase}") + return source.read_range(offset, size) + @property def shape(self) -> tuple[int]: """The shape of the remote array""" diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py new file mode 100644 index 000000000..378a049c2 --- /dev/null +++ b/tests/ndarray/test_c2array_blocks.py @@ -0,0 +1,321 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Block-granular reads of a C2Array, against a stand-in for a subscriber. + +The server here answers the two endpoints the block path uses -- `api/info` for +the geometry and `api/fetch` for the bytes -- the way Caterva2 does: a stored +dataset comes back through a file response that honours `Range`, and one the +subscriber would compute comes back as a stream that ignores it. Which is the +distinction the whole arrangement rests on, and the one thing a test against a +live subscriber could not switch off at will. +""" + +import contextlib +import json +import math +import pathlib +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +import numpy as np +import pytest + +import blosc2 + + +class _Subscriber: + """A Caterva2-shaped server over one .b2nd file.""" + + def __init__(self, path, ranges=True, cookie=None): + self.path = str(path) + self.frame = pathlib.Path(self.path).read_bytes() + self.array = blosc2.open(self.path) + self.ranges = ranges # False: stream the body and ignore Range, as a + self.cookie = cookie # computed dataset does + self.log = [] # (endpoint, status, bytes served) + + @property + def meta(self): + schunk = self.array.schunk + return { + "shape": list(self.array.shape), + "chunks": list(self.array.chunks), + "blocks": list(self.array.blocks), + "dtype": str(self.array.dtype), + "mtime": None, + "schunk": { + "cparams": {"typesize": self.array.dtype.itemsize}, + "nbytes": schunk.nbytes, + "cbytes": schunk.cbytes, + "cratio": schunk.cratio, + "blocksize": schunk.blocksize, + "vlmeta": {}, + }, + } + + +class _Handler(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *args): + pass # no stderr noise per request + + def handle(self): + # A client that hangs up on a body it refused, which is the point of the + # probe, leaves the write half of this raising + with contextlib.suppress(ConnectionResetError, BrokenPipeError): + super().handle() + + def _send(self, status, body, headers=(), endpoint=""): + self.server.subscriber.log.append((endpoint, status, len(body))) + self.send_response(status) + for name, value in headers: + self.send_header(name, value) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + sub = self.server.subscriber + if sub.cookie and self.headers.get("Cookie") != sub.cookie: + self._send(401, b"unauthorized", endpoint="auth") + return + endpoint = self.path.split("/")[2] + if endpoint == "info": + self._send(200, json.dumps(sub.meta).encode(), endpoint="info") + elif endpoint == "chunk": + nchunk = int(self.path.split("nchunk=")[1]) + self._send(200, sub.array.schunk.get_chunk(nchunk), endpoint="chunk") + elif endpoint == "fetch": + self._fetch(sub) + else: + self._send(404, b"", endpoint=endpoint) + + def _fetch(self, sub): + wanted = self.headers.get("Range") + if not wanted or not sub.ranges: + # What a StreamingResponse does with a Range header: nothing at all + self._send(200, sub.frame, endpoint="fetch") + return + start, end = (int(n) for n in wanted.removeprefix("bytes=").split("-")) + end = min(end, len(sub.frame) - 1) + self._send( + 206, + sub.frame[start : end + 1], + [("Content-Range", f"bytes {start}-{end}/{len(sub.frame)}"), ("Accept-Ranges", "bytes")], + endpoint="fetch", + ) + + +def _serve(tmp_path, data, chunks, blocks, ranges=True, cookie=None, name="ds.b2nd"): + """A C2Array over *data*, served by a subscriber stand-in on localhost.""" + urlpath = str(tmp_path / name) + blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") + server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + server.subscriber = _Subscriber(urlpath, ranges=ranges, cookie=cookie) + threading.Thread(target=server.serve_forever, daemon=True).start() + urlbase = f"http://127.0.0.1:{server.server_address[1]}/" + array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=cookie) + return array, server.subscriber, server + + +@pytest.fixture +def subscriber(tmp_path): + """Serve one array; the test parametrizes with `_serve`'s arguments.""" + servers = [] + + def build(*args, **kwargs): + array, sub, server = _serve(tmp_path, *args, **kwargs) + servers.append(server) + return array, sub + + yield build + for server in servers: + server.shutdown() + server.server_close() + + +@pytest.fixture +def any_chunk_wants_blocks(monkeypatch): + """Take the size threshold out of the way, so small test arrays use blocks.""" + monkeypatch.setattr(blosc2.proxy, "BLOCK_MIN_CBYTES", 0) + + +def _incompressible(shape, seed=0): + return np.random.default_rng(seed).random(shape) + + +def _bytes(sub, endpoint): + return sum(n for kind, _, n in sub.log if kind == endpoint) + + +def test_blocks_are_read_over_ranges(subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + # The frame index, then one read for the block offsets and one for the block + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 6 + assert {status for _, status, _ in sub.log} == {206} + assert not _bytes(sub, "chunk") + # A block of a chunk, not the chunk: an eighth of it here, and never the frame + assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 8 + # ... and the rest of the array still arrives correctly afterwards + assert np.array_equal(p[...], data) + + +@pytest.mark.parametrize( + ("shape", "chunks", "blocks", "item"), + [ + ((1000,), (500,), (50,), slice(120, 140)), + ((200, 200), (100, 200), (10, 20), (slice(5, 7), slice(30, 90))), + ((20, 60, 60), (10, 30, 60), (5, 10, 20), (3, slice(10, 20), slice(10, 20))), + ((200, 200), (100, 200), (10, 20), (5, 5)), + ], + ids=["1d", "2d", "3d", "point"], +) +def test_block_reads_are_correct(subscriber, any_chunk_wants_blocks, shape, chunks, blocks, item): + data = _incompressible(shape) + array, _ = subscriber(data, chunks=chunks, blocks=blocks) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[item], data[item]) + assert np.array_equal(p[...], data) + + +def test_blocks_carry_the_auth_cookie(subscriber, any_chunk_wants_blocks): + # fsspec's HTTP filesystem cannot carry this, which is why the block reads of + # a C2Array are its own rather than an fsspec URL pointed at the subscriber + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), cookie="token=sikrit") + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is not None + assert not any(status == 401 for _, status, _ in sub.log) + + +def test_a_streamed_dataset_falls_back_to_chunks(subscriber, any_chunk_wants_blocks): + # A lazy expression, an HDF5 leaf or a .b2z member is built rather than + # stored, and the response that carries it ignores Range + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), ranges=False) + p = blosc2.Proxy(array, mode="w") + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is None + assert [kind for kind, _, _ in sub.log] == ["fetch", "chunk"] + # The probe must not read the body it refused: the whole dataset is what it + # would have downloaded to find out that ranges are not served + assert _bytes(sub, "fetch") == len(sub.frame) # served, but never read + + # And it is never probed again, whatever else is asked for + assert np.array_equal(p[...], data) + assert sum(1 for kind, _, _ in sub.log if kind == "fetch") == 1 + + +def test_a_computed_dataset_is_ruled_out_without_a_request(subscriber, any_chunk_wants_blocks): + # api/info tells a stored dataset from one the subscriber computes: the + # latter reports `expression` and `operands` where this reports a geometry + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + del array.meta["chunks"] + sub.log.clear() + + assert array.block_source() is None + assert not sub.log + + +def test_small_chunks_are_fetched_whole(subscriber): + # Below the threshold a chunk is one cheap request, so blocks would only add + # a round trip: nothing goes looking for the frame index, let alone a block + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is None + assert [kind for kind, _, _ in sub.log] == ["chunk"] + + +def test_blocks_accumulate_in_a_chunk(subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + served = len(sub.log) + # A different block of the same chunk: what is already cached stays cached + assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) + assert len(sub.log) > served + served = len(sub.log) + # Both are now in the same cached chunk, and both are still right + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) + assert len(sub.log) == served + + +def test_blocks_survive_a_reopened_cache(tmp_path, subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "c2-cache.b2nd") + + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + # A partly filled chunk survives, so the blocks in it do not travel again + p = blosc2.Proxy(array, urlpath=cache, mode="a") + served = len(sub.log) + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert len(sub.log) == served + # ... and the ones missing from it still do + assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) + assert len(sub.log) > served + assert np.array_equal(p[...], data) + + +def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): + # A cache left by a run that fetched whole chunks (which is every run before + # this existed) holds complete chunks, so nothing in it is fetched again + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "chunkwise.b2nd") + + array._block_source = None # as if the subscriber served no ranges + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p, array + + array, _ = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, urlpath=cache, mode="a") + served = len(sub.log) + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert len(sub.log) == served + assert np.array_equal(p[...], data) + + +def test_blocks_per_chunk_costs_no_request(subscriber): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + sub.log.clear() + + assert array.blocks_per_chunk == math.prod((100 // 10, 200 // 20)) + assert not sub.log + + +def test_read_range_says_so_when_there_are_no_ranges(subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, _ = subscriber(data, chunks=(100, 200), blocks=(10, 20), ranges=False) + + with pytest.raises(ValueError, match="not served in byte ranges"): + array.read_range(0, 32) From b021104b870e94888d4fcbc02f2324e7c14b35a2 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:53:36 +0200 Subject: [PATCH 05/25] Ask a Caterva2 subscriber for a whole wave of ranges at once RFC 7233 lets a Range header name several spans, and the answer is a multipart/byteranges body carrying each with its own Content-Range. Starlette builds that, so the blocks a slice wants -- across chunks, since they are all the same file -- can travel in one request instead of one each. No object store offers this; it is what makes a subscriber the better backend for block reads rather than merely an equal one. Measured against cat2.cloud on kevlar-tomo.b2nd, a slice touching ten chunks: 20 requests in 0.334 s becomes 2 in 0.204 s, and 1.008 s becomes 0.141 s against a subscriber reached one request at a time. Both waves collapse: the chunk headers that say where the blocks are, then the blocks. The seam is one method. ByteRangeNDSource grows read_ranges(spans), which by default is one read_range each, and max_ranges to say how many a transport will take; C2NDSource overrides the first and raises the second. A source with neither -- every fsspec backend -- is batched one range at a time, which is the reads it always made. The answer is taken apart by what each part says it holds rather than by trusting the order: Starlette sorts the spans and merges the ones that touch, and answers a plain 206 when they all merge into one. A subscriber that answers with less than was asked for is noticed once and never batched again. Also give C2Array a max_concurrency: Proxy.afetch already used 8 for one, and fetch was serial only for want of somewhere to read the figure from. That alone is what takes the unbatched block path from 1.008 s to 0.334 s. Co-Authored-By: Claude Opus 5 --- src/blosc2/c2array.py | 118 +++++++++++++++++++++++++- src/blosc2/proxy.py | 120 ++++++++++++++++++++------- tests/ndarray/test_c2array_blocks.py | 103 +++++++++++++++++++++-- 3 files changed, 299 insertions(+), 42 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index fd7bdbf3c..d631e24ed 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -263,11 +263,63 @@ def slice_to_string(slice_): _UNTRIED = object() """A block source that has not been asked for yet, as against one that failed.""" +MAX_RANGES_PER_REQUEST = 64 +"""How many byte ranges one request to a subscriber may ask for. + +There is no limit in the protocol, and the saving grows with the count -- but a +`Range` header is a header, which servers and proxies cap the length of (8 KB is +the usual figure, and 64 spans of a large frame are about a kilobyte), and a +failed request costs a round trip and every span in it. +""" + class _NotRanged(Exception): """The subscriber answered a range request with something other than a 206.""" +class _PartsMissing(Exception): + """A multi-range answer did not carry all the bytes that were asked for.""" + + +def _content_range(value: str) -> int: + """Where a `Content-Range: bytes start-end/total` header says its part starts.""" + return int(value.split()[1].split("-")[0]) + + +def _byteranges(response) -> list[tuple[int, bytes]]: + """The parts of a 206, as (where each starts in the frame, its bytes). + + One part for an ordinary 206, several for a `multipart/byteranges` body: a + boundary line, the part's own headers, a blank line and its bytes, over and + over, ending in the boundary followed by two dashes. `email` would parse it, + at the price of decoding a megabyte of compressed data as text. + """ + content_type = response.headers.get("content-type", "") + if "multipart/byteranges" not in content_type: + return [(_content_range(response.headers["content-range"]), response.content)] + boundary = content_type.split("boundary=")[1].strip().strip('"').encode() + parts = [] + for chunk in response.content.split(b"--" + boundary): + head, sep, body = chunk.partition(b"\r\n\r\n") + if not sep: # the preamble before the first boundary, and the closing -- + continue + for line in head.split(b"\r\n"): + name, _, value = line.partition(b":") + if name.strip().lower() == b"content-range": + # The body ends with the CRLF that belongs to the next boundary + parts.append((_content_range(value.decode()), body[: body.rfind(b"\r\n")])) + break + return parts + + +def _span_of(parts: list[tuple[int, bytes]], offset: int, size: int, url: str) -> bytes: + """The bytes of one requested span, out of whichever part covers it.""" + for start, data in parts: + if start <= offset < start + len(data): + return data[offset - start : offset - start + size] + raise _PartsMissing(f"{url} answered without the bytes at {offset}, which were asked for") + + class C2NDSource(ByteRangeNDSource): """The frame behind a :ref:`C2Array`, read over HTTP byte ranges. @@ -291,8 +343,40 @@ def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY self._auth_token = array.auth_token super().__init__(self._url, max_concurrency) + max_ranges = MAX_RANGES_PER_REQUEST + def read_range(self, offset: int, size: int) -> bytes: - headers = _auth_headers(self._auth_token, {"Range": f"bytes={offset}-{offset + size - 1}"}) + return self._get([(offset, size)])[0] + + def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: + """Every span in one request, which HTTP has a shape for and S3 has not. + + RFC 7233 lets a `Range` header name several spans, and the answer is a + `multipart/byteranges` body carrying each with its own `Content-Range`. + Starlette builds that, so a whole wave of block reads -- across chunks, + since they are all the same file -- costs one round trip instead of one + each. Measured against cat2.cloud: 32 spans in 0.136 s against 0.208 s + for 32 requests eight at a time, and 1.530 s for them one at a time. + + The server may serve fewer parts than were asked for: Starlette sorts the + spans and merges the ones that touch, and answers a single 206 when they + all merge into one. So the answer is taken apart by what each part says + it holds, and each span read out of the part that covers it, rather than + by trusting the order. A server that answers a multi-range request with + bytes that do not cover the whole of it is not one to ask again: this + keeps to a range per request from there on. + """ + spans = list(spans) + if len(spans) > 1 and self.max_ranges > 1: + try: + return self._get(spans) + except _PartsMissing: + self.max_ranges = 1 + return [self.read_range(*span) for span in spans] + + def _get(self, spans: list[tuple[int, int]]) -> list[bytes]: + wanted = ", ".join(f"{offset}-{offset + size - 1}" for offset, size in spans) + headers = _auth_headers(self._auth_token, {"Range": f"bytes={wanted}"}) with _sync_client().stream("GET", self._url, headers=headers) as response: if response.status_code != 206: # Whatever this is, it is not the bytes that were asked for: a 200 @@ -300,12 +384,23 @@ def read_range(self, offset: int, size: int) -> bytes: # avoid, so leave the body unread on the socket raise _NotRanged(f"{self._url} answered {response.status_code} to a Range request") response.read() - return response.content + parts = _byteranges(response) + return [_span_of(parts, offset, size, self._url) for offset, size in spans] class C2Array(blosc2.Operand): """Remote compressed NDArray accessed from a Caterva2 server.""" + max_concurrency = REMOTE_MAX_CONCURRENCY + """How many fetches a :ref:`Proxy` over this array may run at once. + + Every chunk or block is a request whose cost is mostly the round trip, so + overlapping them is what a remote source has to gain; :meth:`Proxy.afetch` + already used this figure for a `C2Array`, and `fetch` was serial only for + want of somewhere to read it from. `get_chunk` and the range reads are + thread-safe: they share one pooled HTTP client and hold no state of their own. + """ + def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | None = None): """Create an instance of a remote NDArray. @@ -615,20 +710,37 @@ def wants_blocks(self, nchunk: int, nwanted: int) -> bool: source = self.block_source() return source is not None and source.wants_blocks(nchunk, nwanted) + @property + def max_ranges(self) -> int: + """How many ranges one request to this subscriber may carry.""" + source = self.block_source() + return 1 if source is None else source.max_ranges + def chunk_layout(self, nchunk: int): """Where the blocks of a chunk are; see :meth:`ByteRangeNDSource.chunk_layout`.""" return self.block_source().chunk_layout(nchunk) + def chunk_layouts(self, nchunks: Sequence[int]) -> list: + """The same for several chunks; see :meth:`ByteRangeNDSource.chunk_layouts`.""" + return self.block_source().chunk_layouts(nchunks) + def block_plan(self, nchunk: int, nblocks: Sequence[int]) -> list[tuple[int, int, tuple]]: """The range reads covering *nblocks*; see :meth:`ByteRangeNDSource.block_plan`.""" return self.block_source().block_plan(nchunk, nblocks) def read_range(self, offset: int, size: int) -> bytes: """The bytes at [*offset*, *offset* + *size*) of the remote frame.""" + return self._ranged().read_range(offset, size) + + def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: + """The bytes of every span, in one request where the subscriber allows it.""" + return self._ranged().read_ranges(spans) + + def _ranged(self) -> C2NDSource: source = self.block_source() if source is None: raise ValueError(f"{self.path} is not served in byte ranges by {self.urlbase}") - return source.read_range(offset, size) + return source @property def shape(self) -> tuple[int]: diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 3448fcb3e..4a2027138 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -83,6 +83,12 @@ class ProxyNDSource(ABC): ``chunk_layout(nchunk)``, ``block_plan(nchunk, nblocks)`` and ``read_range(offset, size)``; :ref:`FsspecNDSource` implements them over byte ranges and is the worked example. + + A source whose transport can ask for several ranges at once says so with + ``max_ranges`` and serves ``read_ranges(spans)`` and + ``chunk_layouts(nchunks)`` as well; :ref:`Proxy` then sends a whole wave of + reads as one request. Both are optional, and a source without them is asked + one range at a time exactly as before. """ @property @@ -679,6 +685,10 @@ def _fetch_by_block(self, item, max_concurrency: int | None): whatever chunks are cheaper to take whole. Which is which is decided before any of it, so the chunks that do not want blocks cost exactly one request each, as they did before this existed. + + A transport that carries several ranges per request (`max_ranges`) + collapses each wave further, into as few requests as its limit allows; + one that does not sees exactly the reads it always saw. """ missing = self._missing_blocks(item) if not missing: @@ -686,39 +696,56 @@ def _fetch_by_block(self, item, max_concurrency: int | None): wanted = {n: bs for n, bs in missing.items() if self.src.wants_blocks(n, len(bs))} whole = [n for n in missing if n not in wanted] - layouts = dict( - zip(wanted, self._run(self.src.chunk_layout, list(wanted), max_concurrency), strict=True) - ) + layouts = dict(zip(wanted, self._chunk_layouts(list(wanted), max_concurrency), strict=True)) # A chunk with nothing to take apart (memcpyed, or a single block) says so # only once its header is read whole += [n for n, layout in layouts.items() if layout is None] wanted = {n: bs for n, bs in wanted.items() if layouts[n] is not None} - tasks = [(n, None) for n in whole] - tasks += [(n, run) for n in wanted for run in self.src.block_plan(n, wanted[n])] + # Each task is what one request will carry: a whole chunk on its own, or + # a batch of range reads (of one, for a transport that takes one) + runs = [(n, run) for n in wanted for run in self.src.block_plan(n, wanted[n])] + batch = max(getattr(self.src, "max_ranges", 1), 1) + tasks = [((n, None),) for n in whole] + list(itertools.batched(runs, batch)) - def fetch_one(task): - nchunk, run = task - return self.src.get_chunk(nchunk) if run is None else self.src.read_range(run[0], run[1]) + # `read_ranges` is the optional half of the protocol: a source that only + # has `read_range` is asked one range at a time, as `batch` is 1 for it + read_ranges = getattr(self.src, "read_ranges", None) + + def fetch(task): + if task[0][1] is None: + return [self.src.get_chunk(task[0][0])] + spans = [(run[0], run[1]) for _, run in task] + if read_ranges is None: + return [self.src.read_range(*span) for span in spans] + return read_ranges(spans) pending = {} try: - for (nchunk, run), data in zip(tasks, self._run(fetch_one, tasks, max_concurrency), strict=True): - if run is None: - self._store_chunk(nchunk, data) - continue - payloads = pending.setdefault(nchunk, {}) - for nblock, offset, size in run[2]: - payloads[nblock] = data[offset : offset + size] - # Write the chunk once its last outstanding block has landed, so - # nothing is held longer than it takes to splice it in - if len(payloads) == len(wanted[nchunk]): - self._write_blocks(nchunk, pending.pop(nchunk), layouts[nchunk][0]) + for task, answers in zip(tasks, self._run(fetch, tasks, max_concurrency), strict=True): + for (nchunk, run), data in zip(task, answers, strict=True): + if run is None: + self._store_chunk(nchunk, data) + continue + payloads = pending.setdefault(nchunk, {}) + for nblock, offset, size in run[2]: + payloads[nblock] = data[offset : offset + size] + # Write the chunk once its last outstanding block has landed, so + # nothing is held longer than it takes to splice it in + if len(payloads) == len(wanted[nchunk]): + self._write_blocks(nchunk, pending.pop(nchunk), layouts[nchunk][0]) finally: self._save_fetched() return self._cache + def _chunk_layouts(self, nchunks: list[int], max_concurrency: int | None): + """Where the blocks of every one of *nchunks* are, in as few requests as fit.""" + if max(getattr(self.src, "max_ranges", 1), 1) > 1: + # The source batches the reads itself, so there is nothing to overlap + return self.src.chunk_layouts(nchunks) + return self._run(self.src.chunk_layout, nchunks, max_concurrency) + def _write_blocks(self, nchunk: int, payloads: dict[int, bytes], header: bytes) -> None: """Put the blocks just fetched into the cache, keeping those already there. @@ -1196,6 +1223,15 @@ class ByteRangeNDSource(ProxyNDSource): stamp = None + max_ranges = 1 + """How many ranges one request of this transport may carry. + + One means one request each, which is all any object store offers. A + subscriber answering ``multipart/byteranges`` takes more -- see + :meth:`read_ranges` -- and then a slice costs a couple of requests rather + than a couple per chunk it touches. + """ + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self.max_concurrency = max_concurrency self.urlpath = urlpath @@ -1270,6 +1306,16 @@ def read_range(self, offset: int, size: int) -> bytes: overlap the fetches of one slice. """ + def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: + """The bytes of every ``(offset, size)`` in *spans*, in that order. + + One request each, unless a transport that can carry several ranges in one + overrides this and raises :attr:`max_ranges` to say how many. Nothing + else has to change for it: this is the only method a batching transport + needs, and everything that reads bytes goes through it. + """ + return [self.read_range(offset, size) for offset, size in spans] + def wants_blocks(self, nchunk: int, nwanted: int) -> bool: """Whether fetching *nwanted* blocks of a chunk beats fetching all of it. @@ -1307,13 +1353,28 @@ def chunk_layout(self, nchunk: int) -> tuple[bytes, np.ndarray, np.ndarray] | No Each of those is then fetched whole, at the cost of the one header read that found out. """ - cached = self._layouts.get(nchunk) - if cached is not None: - return cached + return self.chunk_layouts((nchunk,))[0] + + def chunk_layouts(self, nchunks: Sequence[int]) -> list: + """:meth:`chunk_layout` for several chunks, in as few requests as they fit. + + One request each unless the transport takes several ranges at once, and + none at all for a chunk already read: a fetch asks for layouts only where + blocks are missing, but the same chunk comes up again as a slice fills it + in. + """ + section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk + todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] + for batch in itertools.batched(todo, max(self.max_ranges, 1)): + spans = [(int(self._offsets[n]), section) for n in batch] + heads = self.read_ranges(spans) + for nchunk, head in zip(batch, heads, strict=True): + self._layouts[nchunk] = self._parse_layout(head, section) + return [self._layouts[n] for n in nchunks] + + def _parse_layout(self, head: bytes, section: int): + """The layout a chunk's header section says it has, or None for no layout.""" nblocks = self.blocks_per_chunk - offset = int(self._offsets[nchunk]) - section = _CHUNK_HEADER_LEN + 4 * nblocks - head = self.read_range(offset, section) cbytes = struct.unpack(" tuple[bytes, np.ndarray, np.ndarray] | No or cbytes < section or len(head) < section ): - layout = None - else: - bstarts = np.frombuffer(head[_CHUNK_HEADER_LEN:], dtype=" list[tuple[int, int, tuple]]: """The range reads that cover *nblocks*, near-adjacent ones merged. diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 378a049c2..46c2d384b 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -31,12 +31,14 @@ class _Subscriber: """A Caterva2-shaped server over one .b2nd file.""" - def __init__(self, path, ranges=True, cookie=None): + def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges=True): self.path = str(path) self.frame = pathlib.Path(self.path).read_bytes() self.array = blosc2.open(self.path) self.ranges = ranges # False: stream the body and ignore Range, as a self.cookie = cookie # computed dataset does + self.multipart = multipart # False: answer only the first range asked for + self.merge_ranges = merge_ranges # as Starlette does with ranges that touch self.log = [] # (endpoint, status, bytes served) @property @@ -102,25 +104,57 @@ def _fetch(self, sub): # What a StreamingResponse does with a Range header: nothing at all self._send(200, sub.frame, endpoint="fetch") return - start, end = (int(n) for n in wanted.removeprefix("bytes=").split("-")) - end = min(end, len(sub.frame) - 1) + spans = [] + for span in wanted.removeprefix("bytes=").split(","): + start, end = (int(n) for n in span.split("-")) + spans.append((start, min(end, len(sub.frame) - 1))) + # Starlette sorts the spans and merges the ones that touch, so a client + # cannot count on getting a part per span, nor on the order it asked in + spans.sort() + merged = [spans[0]] + for start, end in spans[1:]: + if start <= merged[-1][1] + 1 and sub.merge_ranges: + merged[-1] = (merged[-1][0], max(merged[-1][1], end)) + else: + merged.append((start, end)) + if len(merged) == 1 or not sub.multipart: + start, end = merged[0] # ... and answers a plain 206 when one is left + self._send( + 206, + sub.frame[start : end + 1], + [("Content-Range", f"bytes {start}-{end}/{len(sub.frame)}"), ("Accept-Ranges", "bytes")], + endpoint="fetch", + ) + return + boundary = "c2boundary" + body = b"" + for start, end in merged: + body += ( + f"--{boundary}\r\nContent-Type: application/octet-stream\r\n" + f"Content-Range: bytes {start}-{end}/{len(sub.frame)}\r\n\r\n" + ).encode() + body += sub.frame[start : end + 1] + b"\r\n" + body += f"--{boundary}--\r\n".encode() self._send( 206, - sub.frame[start : end + 1], - [("Content-Range", f"bytes {start}-{end}/{len(sub.frame)}"), ("Accept-Ranges", "bytes")], + body, + [ + ("Content-Type", f"multipart/byteranges; boundary={boundary}"), + ("Accept-Ranges", "bytes"), + ], endpoint="fetch", ) -def _serve(tmp_path, data, chunks, blocks, ranges=True, cookie=None, name="ds.b2nd"): +def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", **kwargs): """A C2Array over *data*, served by a subscriber stand-in on localhost.""" urlpath = str(tmp_path / name) blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) - server.subscriber = _Subscriber(urlpath, ranges=ranges, cookie=cookie) + server.subscriber = _Subscriber(urlpath, **kwargs) threading.Thread(target=server.serve_forever, daemon=True).start() urlbase = f"http://127.0.0.1:{server.server_address[1]}/" - array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=cookie) + array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=kwargs.get("cookie")) return array, server.subscriber, server @@ -319,3 +353,56 @@ def test_read_range_says_so_when_there_are_no_ranges(subscriber, any_chunk_wants with pytest.raises(ValueError, match="not served in byte ranges"): array.read_range(0, 32) + + +def test_a_whole_wave_travels_in_one_request(subscriber, any_chunk_wants_blocks): + # A column through every chunk: each one wants a handful of blocks that lie + # apart in the file, which without batching is a request each + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + assert array.block_source() is not None # the frame index, read once per array + sub.log.clear() + + assert np.array_equal(p[:, 0:10], data[:, 0:10]) + # One request for the layouts of all four chunks, one for all their blocks + assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch"] + assert {status for _, status, _ in sub.log} == {206} + assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 4 + + # Which is the whole of the difference: one request per range otherwise + other, sub2 = subscriber(data, chunks=(100, 200), blocks=(10, 20), name="unbatched.b2nd") + other.block_source().max_ranges = 1 + q = blosc2.Proxy(other, mode="w") + sub2.log.clear() + assert np.array_equal(q[:, 0:10], data[:, 0:10]) + assert len(sub2.log) > 4 * len(sub.log) + + +def test_merged_and_reordered_parts_are_read_correctly(subscriber, any_chunk_wants_blocks): + # The server sorts the spans and merges the ones that touch, so the answer + # carries fewer parts than were asked for and in an order of its own + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[:, 0:10], data[:, 0:10]) + assert np.array_equal(p[...], data) + assert array.max_ranges > 1 # ... and it never had to stop batching + + +def test_a_server_that_answers_one_range_stops_being_batched(subscriber, any_chunk_wants_blocks): + # A subscriber that takes the first span of a multi-range request and ignores + # the rest: the answer does not cover what was asked for, which is noticed + # and never repeated + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), multipart=False) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[:, 0:10], data[:, 0:10]) + assert array.max_ranges == 1 + served = len(sub.log) + assert np.array_equal(p[:, 100:110], data[:, 100:110]) + # One request per range from here on, and no second attempt at batching + assert len(sub.log) > served + 2 + assert np.array_equal(p[...], data) From 1b2dd44769947a84bcca15cc8a00bbba8d1d1ad8 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:54:36 +0200 Subject: [PATCH 06/25] Record what the block-granularity work landed as, and what it found The plan said an HDF5 leaf or a .b2z member might report a geometry while being streamed, and that a C2Array over a computed dataset would be caught by the api/info check. Both turned out slightly otherwise, and the plan now says so rather than reading as if it had been right. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 15 ++++++ plans/cat2-block-granularity.md | 92 +++++++++++++++++++++++++++++++-- 2 files changed, 104 insertions(+), 3 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 165923a66..eea232aec 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -42,6 +42,21 @@ XXX version-specific blurb XXX with the traffic, since a fetch in flight is now a block rather than a chunk. `bench/ndarray/fsspec-block-granularity.py` measures both on any array. +* A `Proxy` over a `C2Array` reads **blocks** too, straight out of the stored + frame over HTTP byte ranges. Caterva2 serves a stored dataset from a file, so + the `Range` header is honoured and composes with the auth cookie; no new + endpoint is involved. On cat2.cloud's `kevlar-tomo.b2nd` a corner slice costs + 0.031 MB instead of 2.723 MB, and a slice touching ten chunks takes 0.14 s + against 1.01 s. Three things add up to it: one pooled HTTP client instead of a + connection per request (0.162 s → 0.046 s each), fetches that overlap by + default as `afetch()` already did, and one request carrying the whole wave of + ranges (`multipart/byteranges`, which no object store offers). A dataset the + subscriber *computes* — a lazy expression, an HDF5 leaf, a `.b2z` member — is + fetched a whole chunk at a time as before; which it is costs at most one small + request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is + the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it + with a `read_range(offset, size)` to give any transport the same treatment. + * `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an earlier run instead of failing on the existing file, so a proxy's cache can outlive the process. The cache must come from a proxy over a source of the same diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 4e6addb51..e0d038fea 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -1,8 +1,12 @@ # Block-Granular Reads For Caterva2 And `C2Array` -Analysis and plan — nothing implemented. Written 2026-08-17, after -[plans/fsspec-blocks.md](fsspec-blocks.md) landed block fetching for fsspec URLs -(merged as PR #701). +Written 2026-08-17, after [plans/fsspec-blocks.md](fsspec-blocks.md) landed block +fetching for fsspec URLs (merged as PR #701). + +**All five phases are implemented** (2026-08-17). Phases 1-4 are in blosc2 on +`cat2-block-granularity`; phase 5 is in Caterva2 on `range-honesty`. See +[what landed](#what-landed) at the end for the results and the two things the +work found out. ## The question @@ -312,3 +316,85 @@ print(r.status_code, r.headers.get("content-range"), len(r.content)) `bench/ndarray/fsspec-block-granularity.py` measures the touch ratios of any local array, which is what decides whether a given dataset would benefit at all. + +## What landed + +Every phase, in the order the plan gives them. Where the plan guessed and the +work found out otherwise, that is said below rather than quietly fixed. + +| phase | where | commit | +|---|---|---| +| 1. pooled client | blosc2 `c2array.py` | *Pool the HTTP client C2Array requests go through* | +| 2. capability check | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | +| 3b. `ByteRangeNDSource` | blosc2 `proxy.py` | *Lift the frame reading out of FsspecNDSource* | +| 3. `C2Array` blocks | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | +| 4. multipart | blosc2 both | *Ask a subscriber for a whole wave of ranges at once* | +| 5. honest streaming | caterva2 `server.py` | *Say which responses serve byte ranges* | + +Option **(b)** was taken for phase 3, as recommended: `ByteRangeNDSource` holds +the frame format and one abstract `read_range`, `FsspecNDSource` is that plus +four lines of fsspec, and `C2NDSource` is that plus HTTP ranges with the auth +cookie. `C2Array` keeps the five members `Proxy` looks for and delegates them, +so every existing `Proxy(C2Array(...))` gets blocks without being asked. + +### The numbers, end to end + +Against `cat2.cloud/demo` on `kevlar-tomo.b2nd` (1.44 MB chunks, 47 blocks each): + +| | requests | bytes | time | +|---|---|---|---| +| chunks (before) | 2 | 2.723 MB | 0.28 s | +| blocks | 8 | 0.031 MB | 0.34 s | + +88x fewer bytes for a corner slice, and the same wall time on a link where a +round trip and a megabyte cost about the same. For a slice touching ten chunks, +where the request count is what decides: + +| | requests | time | +|---|---|---| +| blocks, one request at a time | 20 | 1.008 s | +| blocks, fetches overlapped | 20 | 0.334 s | +| blocks, multipart | 2 | 0.141 s | + +Phase 1 on its own: 0.162 s per request against 0.046 s pooled, on the existing +chunk path. + +### Two things the plan had wrong + +- **`C2Array` cannot be built over a computed dataset at all.** `api/info` for a + lazy expression carries no `schunk`, and `C2Array.__init__` reads + `meta["schunk"]["cparams"]`, so it raises long before any of this. The + info-based discriminator of phase 2 is still there and still right, but it + earns its place on the *other* case: +- **A `.b2z` member reports a full geometry and is streamed.** Confirmed against + a local server: `api/info` on `@public/tree-store.b2z/level1/leaf6` answers + with `blocks`, `chunks` and `schunk`, and `api/fetch` streams it. So the + status code of the first range read is the authority, exactly as phase 2 + argued — with phase 5 in place that costs 169 bytes and one round trip. + +Phase 4 was built because the measurement said to: 32 spans cost 0.136 s in one +multipart request against 0.208 s as 32 requests eight at a time, and 1.530 s +one at a time. Starlette *sorts and merges* the spans it is given and answers a +plain 206 when they all merge into one, so the client maps parts back by what +each says it holds rather than by order; a server that answers with less than +was asked for is noticed once and never batched again. + +The plan's guess that `_run` would already overlap a C2Array's fetches was wrong +in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, and +`C2Array` had none, so the sync path was serial. It has one now, the same 8 +`afetch` already used. + +### Left undone + +- **`api/chunk` does not serve container members.** A `Proxy` over a `.b2z` leaf + falls back to whole chunks correctly and then 404s, because `get_chunk` in the + server resolves the path without an inner key. It has never worked; nothing + here changed it, and nothing here depends on it. +- **The four requests to open a frame** (prefix, header, offsets header, offsets) + could be two: the header could be read optimistically with the prefix. It is in + `ByteRangeNDSource`, so it would pay for fsspec as well, and multipart would + make it one. +- **`C2Array` still has no `stamp`.** A cache is checked against the source's + geometry only, so a dataset replaced underneath while keeping its shape is not + noticed. `mtime` is in `api/info` and would do it, but adding one invalidates + every cache built before it, which wants its own decision. From b868c53f0e1685acb85f282cd03d61b3eb387fd6 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:56:58 +0200 Subject: [PATCH 07/25] Keep C2NDSource's max_ranges with the rest of its class body --- src/blosc2/c2array.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index d631e24ed..d2c1649dc 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -338,13 +338,13 @@ class C2NDSource(ByteRangeNDSource): the array, whose ``api/chunk`` path works for every dataset there is. """ + max_ranges = MAX_RANGES_PER_REQUEST + def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self._url = _sub_url(array.urlbase, f"api/fetch/{array.path}") self._auth_token = array.auth_token super().__init__(self._url, max_concurrency) - max_ranges = MAX_RANGES_PER_REQUEST - def read_range(self, offset: int, size: int) -> bytes: return self._get([(offset, size)])[0] From e8d9ba1a4caab3647d6f47b174469c20fc6bc685 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:15:11 +0200 Subject: [PATCH 08/25] Measure the Caterva2 block path the way the fsspec one is measured bench/ndarray/fsspec-block-granularity.py answers "would blocks beat chunks for this array, and by how much" for an fsspec URL. This is the same question for a Caterva2 dataset, where it has a different shape: whether the dataset serves ranges at all is the first thing to know, and three separate changes -- a pooled client, overlapped fetches, one request carrying many ranges -- decide the rest. It prints, for a dataset of the caller's choosing: whether ranges are served and what finding that out cost, what the frame index costs, the request plan of each mode for six slice shapes, what a pooled connection is worth against a client per request, and the timed slices themselves. The plans come from Proxy._wanted_blocks and the source's own block_plan rather than from a reimplementation, and the chunk sizes are read out of the chunk headers rather than guessed from the gaps between them, so the counts in the plan table are the ones the timed run then produces. No service is needed: a stand-in subscriber serves api/info, api/fetch and api/chunk over loopback from any local .b2nd, with ranges, multipart, and the sort-and-merge Starlette does to the spans it is given. --streamed makes it answer the way a computed dataset is served, which is what the fallback costs. --latency-ms and --bandwidth-mbs put a network back in front of loopback. Against cat2.cloud/demo on kevlar-tomo.b2nd, a line through all ten chunks: chunk mode 10 req 14.43 MB 1.008 s blocks 20 req 0.20 MB 0.202 s multipart 2 req 0.20 MB 0.107 s 9.4x and the two slabs that want every block of their chunk come out identical in all three modes, which is the threshold declining to take a chunk apart. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 2 + bench/ndarray/cat2-block-granularity.py | 550 ++++++++++++++++++++++++ plans/cat2-block-granularity.md | 14 +- 3 files changed, 564 insertions(+), 2 deletions(-) create mode 100644 bench/ndarray/cat2-block-granularity.py diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index eea232aec..d153146c2 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -56,6 +56,8 @@ XXX version-specific blurb XXX request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it with a `read_range(offset, size)` to give any transport the same treatment. + `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, + against a real subscriber or a stand-in it starts itself. * `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an earlier run instead of failing on the existing file, so a proxy's cache can diff --git a/bench/ndarray/cat2-block-granularity.py b/bench/ndarray/cat2-block-granularity.py new file mode 100644 index 000000000..e1c9e0f92 --- /dev/null +++ b/bench/ndarray/cat2-block-granularity.py @@ -0,0 +1,550 @@ +#!/usr/bin/env python + +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Would block-granular reads beat whole-chunk ones for a given Caterva2 dataset? + +A ``Proxy`` over a ``C2Array`` used to fetch one whole compressed chunk per +request, through ``api/chunk``. A chunk is made of blocks, which blosc2 +compresses and decompresses independently, and Caterva2 serves a *stored* +dataset straight from its file, so ``api/fetch`` honours a ``Range`` header and +a slice can fetch only the blocks it touches. Whether that pays depends on +three things this script measures against a subscriber of your choosing: + +- whether the dataset **serves ranges at all**. One stored from a file does; + one the subscriber computes -- a lazy expression, an HDF5 leaf, a ``.b2z`` + member -- is streamed, cannot honour a range, and keeps to whole chunks; +- the **request plan**: how many requests each mode issues and how many bytes + they carry, read from the frame's own chunk headers for a few hundred bytes; +- the **wall time** of each pattern, which is the shipped code fetching real + slices from a real server. + +Usage +----- + # a local array, served by a stand-in subscriber over loopback + python cat2-block-granularity.py mydata.b2nd + + # ... with a network put back in front of every request + python cat2-block-granularity.py mydata.b2nd --latency-ms 45 --bandwidth-mbs 10 + + # ... served the way a computed dataset is, which is what the fallback costs + python cat2-block-granularity.py mydata.b2nd --streamed + + # against a real subscriber + python cat2-block-granularity.py @public/examples/kevlar-tomo.b2nd \\ + --urlbase https://cat2.cloud/demo + + # ... an authenticated dataset + python cat2-block-granularity.py @personal/mine.b2nd --urlbase http://localhost:8000 \\ + --username me@example.com --password foobar11 + +Three modes are timed, each of them the shipped code with one thing changed: + +- ``chunks``: one ``api/chunk`` request per touched chunk. What a proxy over a + C2Array did before blocks existed, and what it still does for a dataset that + cannot serve ranges or whose chunks are too small to be worth taking apart. +- ``blocks``: one request for the block offsets of each chunk worth taking + apart, then one per coalesced run of the blocks wanted. Two dependent waves. +- ``multipart``: the same two waves, each collapsed into a single request -- + RFC 7233 lets one ``Range`` header name many spans, and Caterva2 answers + ``multipart/byteranges``. No object store offers this. + +The stand-in subscriber answers ``api/info``, ``api/fetch`` and ``api/chunk`` +the way Caterva2 does, ranges and multipart included (it sorts and merges the +spans it is given, as Starlette does, which is what the client has to survive). +Its request and byte counts are exact. Its *times* are not a subscriber's: +loopback answers in a fraction of a millisecond, where a subscriber over a WAN +takes tens of milliseconds, which is the regime the whole trade lives in. +``--latency-ms`` and ``--bandwidth-mbs`` put a stated network back in front of +each request; cat2.cloud from Europe measures about ``--latency-ms 45 +--bandwidth-mbs 10``. The simulated bandwidth is *per request*, so eight +parallel ones get eight times as much of it -- which is about right for an +object store and about wrong for one subscriber, and is why ``multipart`` can +come out behind ``blocks`` there while it wins against the real thing. + +Bytes counted are payload: the multipart envelope (about a hundred bytes per +part) and the HTTP headers of every request are not in them. +""" + +import argparse +import http.server +import json +import math +import pathlib +import statistics +import struct +import threading +import time + +import blosc2 +from blosc2 import c2array + +CHUNK_HEADER = blosc2.proxy._CHUNK_HEADER_LEN + + +# +# A stand-in subscriber, so this runs with no service to point at +# + + +class Subscriber: + """Caterva2's three read endpoints over one local .b2nd file.""" + + def __init__(self, urlpath, streamed=False): + self.path = pathlib.Path(urlpath) + self.size = self.path.stat().st_size + self.array = blosc2.open(str(self.path)) + # A dataset the subscriber would compute rather than store: served by a + # body builder, which has no way to honour a Range + self.streamed = streamed + + def meta(self): + schunk = self.array.schunk + return { + "shape": list(self.array.shape), + "chunks": list(self.array.chunks), + "blocks": list(self.array.blocks), + "dtype": str(self.array.dtype), + "mtime": None, + "schunk": { + "cparams": {"typesize": self.array.dtype.itemsize}, + "nbytes": schunk.nbytes, + "cbytes": schunk.cbytes, + "cratio": schunk.cratio, + "blocksize": schunk.blocksize, + "vlmeta": {}, + }, + } + + def read(self, start, end): + """The bytes at [start, end], seeked to rather than materialized.""" + with self.path.open("rb") as frame: + frame.seek(start) + return frame.read(end - start + 1) + + +class Handler(http.server.BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + BOUNDARY = "c2boundary" + + def log_message(self, *args): + pass + + def _send(self, status, body, headers=()): + self.send_response(status) + for name, value in headers: + self.send_header(name, value) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): # noqa: N802 (BaseHTTPRequestHandler's own spelling) + sub = self.server.subscriber + endpoint = self.path.split("/")[2] + if endpoint == "info": + self._send(200, json.dumps(sub.meta()).encode()) + elif endpoint == "chunk": + nchunk = int(self.path.split("nchunk=")[1]) + self._send(200, sub.array.schunk.get_chunk(nchunk)) + elif endpoint == "fetch": + self._fetch(sub) + else: + self._send(404, b"") + + def _fetch(self, sub): + wanted = self.headers.get("Range") + if sub.streamed: + # What the streaming paths answer since they were made honest: a 416 + # instead of the whole body with a 200 that no client could notice + if wanted: + self._send(416, b"", [("Accept-Ranges", "none")]) + else: + self._send(200, sub.read(0, sub.size - 1), [("Accept-Ranges", "none")]) + return + if not wanted: + self._send(200, sub.read(0, sub.size - 1), [("Accept-Ranges", "bytes")]) + return + spans = [] + for span in wanted.removeprefix("bytes=").split(","): + start, end = (int(n) for n in span.split("-")) + spans.append((start, min(end, sub.size - 1))) + # Starlette sorts the spans and merges the ones that touch, and answers a + # plain 206 when only one is left, so a client cannot count on a part per + # span nor on the order it asked in + spans.sort() + merged = [spans[0]] + for start, end in spans[1:]: + if start <= merged[-1][1] + 1: + merged[-1] = (merged[-1][0], max(merged[-1][1], end)) + else: + merged.append((start, end)) + if len(merged) == 1: + start, end = merged[0] + self._send( + 206, + sub.read(start, end), + [("Content-Range", f"bytes {start}-{end}/{sub.size}"), ("Accept-Ranges", "bytes")], + ) + return + body = b"" + for start, end in merged: + body += ( + f"--{self.BOUNDARY}\r\nContent-Type: application/octet-stream\r\n" + f"Content-Range: bytes {start}-{end}/{sub.size}\r\n\r\n" + ).encode() + body += sub.read(start, end) + b"\r\n" + body += f"--{self.BOUNDARY}--\r\n".encode() + self._send( + 206, + body, + [ + ("Content-Type", f"multipart/byteranges; boundary={self.BOUNDARY}"), + ("Accept-Ranges", "bytes"), + ], + ) + + +def stand_in(urlpath, streamed=False): + """Serve *urlpath* as ``@public/``, and return (server, urlbase, path).""" + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + server.subscriber = Subscriber(urlpath, streamed) + threading.Thread(target=server.serve_forever, daemon=True).start() + urlbase = f"http://127.0.0.1:{server.server_address[1]}/" + return server, urlbase, f"@public/{pathlib.Path(urlpath).name}" + + +# +# What each mode would ask for +# + + +def default_patterns(shape): + """Slices worth asking about, for an array of any shape (as in the fsspec bench).""" + mid = [s // 2 for s in shape] + return [ + ("point", tuple(mid)), + ("line, last dim", (*mid[:-1], slice(None))), + ("line, first dim", (slice(None), *mid[1:])), + ( + "window (1/64 per dim)", + tuple(slice(m, m + max(1, s // 64)) for m, s in zip(mid, shape, strict=True)), + ), + ( + "slab (1% of dim 0)", + (slice(mid[0], mid[0] + max(1, shape[0] // 100)), *[slice(None)] * (len(shape) - 1)), + ), + ( + "slab (10% of dim 0)", + (slice(mid[0], mid[0] + max(1, shape[0] // 10)), *[slice(None)] * (len(shape) - 1)), + ), + ] + + +def chunk_cbytes(source, nchunks): + """The compressed size of each chunk, from 16 bytes of its header. + + Read rather than guessed at: the distance to the next chunk is an upper + bound only, and a frame with a hole in it would make the chunk mode look + dearer than it is. One request for the lot where the subscriber takes + several ranges, which is the same trick the fetch path uses. + """ + live = [n for n in nchunks if int(source._offsets[n]) >= 0] + heads = source.read_ranges([(int(source._offsets[n]), 16) for n in live]) if live else [] + sizes = dict.fromkeys(nchunks, 0) # a run-length chunk has no bytes in the file + for nchunk, head in zip(live, heads, strict=True): + sizes[nchunk] = struct.unpack(" 1 else 'no'}" + f" ({source.max_ranges} spans per request)\n" + f" frame index: {opening['requests']} requests, {opening['bytes']} bytes, once per C2Array" + ) + + proxy = blosc2.Proxy(array, mode="w") + print( + f"\n {'pattern':22s} {'chunks':>6s} {'blocks':>13s} {'chunk mode':>20s} " + f"{'block mode':>20s} {'multipart':>9s} ratio" + ) + plans = {} + for name, item in default_patterns(array.shape): + plan = plans[name] = request_plan(proxy, array, item) + print( + f" {name:22s} {plan['chunks touched']:6d} " + f"{plan['blocks wanted']:6d}/{plan['blocks total']:<6d} " + f"{plan['chunk requests']:5d} req {plan['chunk bytes'] / 1e6:7.2f} MB " + f"{plan['block requests']:5d} req {plan['block bytes'] / 1e6:7.2f} MB " + f"{plan['multipart requests']:5d} req {plan['ratio'] * 100:6.1f}%" + ) + + pooled, fresh = connection_setup(urlbase, path, token, args.reps) + print( + f"\n connection setup (api/info): {pooled * 1e3:6.1f} ms pooled, {fresh * 1e3:6.1f} ms with " + f"a client per request ({fresh / pooled:.1f}x)" + ) + _time_patterns(args, open_array, array, ["chunks", "blocks", "multipart"], latency, bandwidth, plans) + + +def _opened(array): + """What reading the frame index cost, which is paid once per C2Array.""" + fresh = c2array.C2Array(array.path, urlbase=array.urlbase, auth_token=array.auth_token) + tally = {"requests": 0, "bytes": 0} + original = c2array.C2NDSource.read_range + + def counted(self, offset, size): + tally["requests"] += 1 # before the read: a refused probe is a request too + data = original(self, offset, size) + tally["bytes"] += len(data) + return data + + c2array.C2NDSource.read_range = counted + try: + fresh.block_source() + finally: + c2array.C2NDSource.read_range = original + return tally + + +def _time_patterns(args, open_array, array, modes, latency, bandwidth, plans=None): + if latency or bandwidth: + print( + f"\n simulating a network: {latency * 1e3:.0f} ms per request" + + (f", {bandwidth / 1e6:.1f} MB/s across it" if bandwidth else "") + ) + header = "".join(f"{mode:>28s}" for mode in modes) + print(f"\n {'pattern':22s}{header}" + (" vs chunks" if len(modes) > 1 else "")) + for name, item in default_patterns(array.shape): + if plans and plans[name]["chunk bytes"] > args.max_mb * 1e6: + print(f" {name:22s} skipped ({plans[name]['chunk bytes'] / 1e6:.0f} MB > --max-mb)") + continue + results = {} + for mode in modes: + runs = [ + timed_slice(open_array, item, mode, args.concurrency, latency, bandwidth) + for _ in range(args.reps) + ] + results[mode] = (statistics.median(r[0] for r in runs), runs[0][1], runs[0][2]) + line = "".join( + f"{results[mode][1]:5d} req {results[mode][2] / 1e6:7.2f} MB {results[mode][0]:6.3f}s" + for mode in modes + ) + speedup = "" + if len(modes) > 1: # what the last mode, which is the shipped one, is worth + speedup = f" {results[modes[0]][0] / results[modes[-1]][0]:8.1f}x" + print(f" {name:22s}{line}{speedup}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index e0d038fea..9753a4acd 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -314,8 +314,18 @@ print(r.status_code, r.headers.get("content-range"), len(r.content)) # 200 None 14435027 -> served by a body builder, blocks must not be used ``` -`bench/ndarray/fsspec-block-granularity.py` measures the touch ratios of any -local array, which is what decides whether a given dataset would benefit at all. +`bench/ndarray/cat2-block-granularity.py` answers all of this for a dataset of +your choosing, and needs no server to point at (it stands one in over loopback): + +```sh +python bench/ndarray/cat2-block-granularity.py mydata.b2nd +python bench/ndarray/cat2-block-granularity.py @public/examples/kevlar-tomo.b2nd \ + --urlbase https://cat2.cloud/demo +``` + +It says whether the dataset serves ranges at all, what each mode would ask for, +and what each one costs. `bench/ndarray/fsspec-block-granularity.py` is the same +question for an fsspec URL. ## What landed From f9f755cc063a8350050ee8308a48b8ce62f005d5 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:20:37 +0200 Subject: [PATCH 09/25] Say when a dataset has no chunks to fetch, rather than tracebacking Run against a .b2z member, the bench died in httpx: api/chunk resolves a path without its inner key and 404s, so a proxy over a container leaf cannot fetch chunks from it at all. That has never worked and is nothing this branch changed, but a tool whose job is to answer "what would this dataset cost" should answer it. Co-Authored-By: Claude Opus 5 --- bench/ndarray/cat2-block-granularity.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/bench/ndarray/cat2-block-granularity.py b/bench/ndarray/cat2-block-granularity.py index e1c9e0f92..7181cc49e 100644 --- a/bench/ndarray/cat2-block-granularity.py +++ b/bench/ndarray/cat2-block-granularity.py @@ -497,6 +497,17 @@ def open_array(): _time_patterns(args, open_array, array, ["chunks", "blocks", "multipart"], latency, bandwidth, plans) +def _no_chunks(array): + """Why api/chunk cannot serve this dataset, if it cannot.""" + import httpx + + try: + array.get_chunk(0) + except httpx.HTTPStatusError as exc: + return f"api/chunk answers {exc.response.status_code} for {array.path}" + return None + + def _opened(array): """What reading the frame index cost, which is paid once per C2Array.""" fresh = c2array.C2Array(array.path, urlbase=array.urlbase, auth_token=array.auth_token) @@ -518,6 +529,13 @@ def counted(self, offset, size): def _time_patterns(args, open_array, array, modes, latency, bandwidth, plans=None): + unavailable = _no_chunks(array) + if unavailable: + # A container member (a .b2z or .h5 leaf) is fetchable but not chunk-wise: + # api/chunk resolves a path without its inner key, so it 404s. A proxy + # over one cannot read it at all, whatever mode it would have used + print(f"\n nothing to time: {unavailable}") + return if latency or bandwidth: print( f"\n simulating a network: {latency * 1e3:.0f} ms per request" From 4fe79af4e4a14140922d5166b4b41cc8f61a713b Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:39:49 +0200 Subject: [PATCH 10/25] The api/chunk gap is fixed; note what a container leaf still lacks Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 9753a4acd..7a939fc7b 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -396,10 +396,19 @@ in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, an ### Left undone -- **`api/chunk` does not serve container members.** A `Proxy` over a `.b2z` leaf - falls back to whole chunks correctly and then 404s, because `get_chunk` in the - server resolves the path without an inner key. It has never worked; nothing - here changed it, and nothing here depends on it. +- ~~**`api/chunk` does not serve container members.**~~ Fixed in Caterva2 on + `range-honesty` (*Serve chunks of a container leaf*): the endpoint resolves the + way `api/fetch` does, so a TreeStore leaf hands over its stored chunk, while + HDF5 leaves and CTables are refused with a 400 naming `slice_` rather than + being recompressed per request. A `.b2z` leaf still gets whole chunks only: + giving one the block path needs the offset of its frame inside the container, + which is the next item. +- **A container leaf could serve ranges too.** A TreeStore keeps its leaves as + ordinary frames inside the `.b2z`, so the bytes a block reader wants are in + the file at a fixed offset -- what is missing is a way for the server to say + where a leaf's frame starts, and for the client to add that base to every + range. Worth its own plan; it would give `.b2z` members everything a plain + `.b2nd` has. - **The four requests to open a frame** (prefix, header, offsets header, offsets) could be two: the header could be read optimistically with the prefix. It is in `ByteRangeNDSource`, so it would pay for fsspec as well, and multipart would From 944ca8004eb672c34364cc8289321e7a0694224d Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:48:55 +0200 Subject: [PATCH 11/25] Open a frame in two requests instead of four The format asks to be read in four steps -- how long is the header, the header, how long is the offsets chunk, the offsets chunk -- and each of the two questions costs a round trip to learn a number smaller than the answer it asks for. Over a network that is half the cost of opening an array. Both are guessed at instead: 8 KB of the head, which holds any ordinary header (165-320 bytes in practice), and the tail that the frame's own length bounds, capped at 64 KB so that a large trailer is not dragged along with the offsets (compressed offsets are small -- 4 KB for a frame of 100_000 chunks). A guess that falls short is followed by the exact read that would have happened anyway, so the worst case is the cost of today. A frame small enough to arrive whole in the first read costs one request: its offsets chunk is in those bytes already. Measured against cat2.cloud on kevlar-tomo.b2nd, opening a C2Array: as the format asks 0.237 s 4 requests, 303 bytes optimistic 0.138 s 2 requests, 8_306 bytes More bytes, fewer round trips, which is the trade worth making: 8 KB is under a millisecond on any link where a round trip is 45. It is in ByteRangeNDSource, so blosc2.open(url, lazy=True) gets it too. The seek/read shim the reader needed goes away with it -- it reads exact ranges now, which is what the transport underneath was doing all along. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 5 ++ plans/cat2-block-granularity.md | 10 ++-- src/blosc2/proxy.py | 79 ++++++++++++++++------------ tests/ndarray/test_c2array_blocks.py | 5 +- tests/test_fsspec.py | 57 ++++++++++++++++++++ 5 files changed, 116 insertions(+), 40 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index d153146c2..79419be93 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -56,6 +56,11 @@ XXX version-specific blurb XXX request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it with a `read_range(offset, size)` to give any transport the same treatment. + Opening a remote frame through either of them now costs two requests instead + of four (0.237 s → 0.138 s against cat2.cloud), and one for a frame small + enough to arrive whole in the first read: the two reads that only measured the + next one are guessed at generously instead, since over a network a few hundred + bytes and a few kilobytes cost the same. `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 7a939fc7b..1c79d2261 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -409,10 +409,12 @@ in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, an where a leaf's frame starts, and for the client to add that base to every range. Worth its own plan; it would give `.b2z` members everything a plain `.b2nd` has. -- **The four requests to open a frame** (prefix, header, offsets header, offsets) - could be two: the header could be read optimistically with the prefix. It is in - `ByteRangeNDSource`, so it would pay for fsspec as well, and multipart would - make it one. +- ~~**The four requests to open a frame**~~ are two, and one for a frame that + arrives whole in the first read (*Open a frame in two requests*): both reads + that only measured the next one are guessed at instead. 0.237 s → 0.138 s + against cat2.cloud. Getting to one for a large frame needs a suffix range + (`Range: bytes=-65536`) batched with the head read, which no fsspec backend + exposes and `read_ranges` has no way to express. - **`C2Array` still has no `stamp`.** A cache is checked against the source's geometry only, so a dataset replaced underneath while keeping its shape is not noticed. `mtime` is in `api/info` and would do it, but adding one invalidates diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 4a2027138..a323acd8e 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -52,6 +52,16 @@ # few unwanted bytes for one less request, which at object-store latencies is # always the right way round. BLOCK_GAP = 4096 +# Opening a frame asks two questions whose answers are shorter than the questions +# are dear: how long is the header, and how long is the offsets chunk. Both are +# guessed at generously instead, since over a network a read of a few hundred +# bytes and one of a few kilobytes cost the same. A guess that falls short costs +# the exact read that would have happened anyway, so these are ceilings, not +# promises: an ordinary header is 165-320 bytes, and a frame of 100_000 chunks +# has 4 KB of compressed offsets. +_FRAME_PREFETCH = 8192 +_INDEX_PREFETCH = 1 << 16 + # How many partly filled chunks keep their blocks in memory as well as in the # cache. Adding a block to a chunk rewrites that chunk, and the blocks already # in it have to come from somewhere: from here, or read back out of the cache and @@ -1101,25 +1111,33 @@ def _chunk_payloads(chunk: bytes, nblocks: int, wanted) -> dict[int, bytes]: return {int(n): chunk[bstarts[n] : bstarts[n] + extents[n]] for n in wanted} -def _read_frame_index(f) -> tuple[bytes, list, np.ndarray]: - """Read the header and the chunk offsets of a contiguous frame from *f*. +def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: + """Read the header and the chunk offsets of a contiguous frame. + + *read_range* is ``(offset, size) -> bytes``, so what this costs is round + trips. Returns the raw header bytes, the header decoded as the msgpack + array it is, and the absolute position of every chunk. A negative position + is not a position at all: it encodes a run-length chunk that was never + written to the file. See ``README_CFRAME_FORMAT.rst`` in c-blosc2 for the + layout. - Returns the raw header bytes, the header decoded as the msgpack array it is, - and the absolute position of every chunk. A negative position is not a - position at all: it encodes a run-length chunk that was never written to the - file. See ``README_CFRAME_FORMAT.rst`` in c-blosc2 for the layout. + The format asks to be read in four steps -- how long is the header, the + header, how long is the offsets chunk, the offsets chunk -- and each of the + two questions costs as much as the answer it asks for. So both are guessed + at instead: enough of the head to hold any ordinary header, and the tail + that the frame's own length bounds. A guess that falls short is followed by + the exact read that would have happened anyway, and a frame small enough to + arrive whole in the first read costs one. """ import msgpack - f.seek(0) - prefix = f.read(24) - if prefix[2:10] != _FRAME_MAGIC: + head = read_range(0, _FRAME_PREFETCH) + if head[2:10] != _FRAME_MAGIC: raise ValueError("not a Blosc2 contiguous frame") # header_len is the one field that must be located by hand; everything after # it comes out of unpacking the header, which is plain msgpack - header_len = struct.unpack(">i", prefix[11:15])[0] - f.seek(0) - raw = f.read(header_len) + header_len = struct.unpack(">i", head[11:15])[0] + raw = head[:header_len] if header_len <= len(head) else read_range(0, header_len) # raw=True because the flags field is a msgpack *string* holding four raw # bytes, and codec_flags packs clevel into its high nibble: from clevel 8 up # that byte is not valid UTF-8 and decoding the header blows up @@ -1130,31 +1148,24 @@ def _read_frame_index(f) -> tuple[bytes, list, np.ndarray]: if header[8] == 0: # chunksize return raw, header, np.empty(0, dtype=np.int64) - # The offsets live in a Blosc2 chunk of their own, right after the data ones - index_pos = header[1] + header[5] - f.seek(index_pos) - index_cbytes = struct.unpack("= frame_len: + index = head[index_pos:] # the whole frame arrived in the first read + else: + index = read_range(index_pos, min(frame_len - index_pos, _INDEX_PREFETCH)) + index_cbytes = struct.unpack(" len(index): + index = read_range(index_pos, index_cbytes) + offsets = np.frombuffer(blosc2.decompress2(index[:index_cbytes]), dtype=np.int64) # Offsets are relative to the end of the header return raw, header, np.where(offsets >= 0, offsets + header_len, offsets) -class _RangeReader: - """The seek/read pair `_read_frame_index` needs, served by exact range requests.""" - - def __init__(self, read_range): - self._read_range, self._pos = read_range, 0 - - def seek(self, pos: int) -> None: - self._pos = pos - - def read(self, size: int) -> bytes: - data = self._read_range(self._pos, size) - self._pos += len(data) - return data - - def _frame_metalayer(raw: bytes, header: list, name: str): """Decode the *name* metalayer out of an already-read frame header.""" offset = header[13][1][name.encode()] # KeyError if there is no such metalayer @@ -1238,7 +1249,7 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): # Exact ranges, not a file handle: a buffered one reads a whole block per # seek (50 MiB on s3fs by default), which would undo the point of a lazy # open. Chunk reads are stateless, so nothing here is shared between threads - raw, header, self._offsets = _read_frame_index(_RangeReader(self.read_range)) + raw, header, self._offsets = _read_frame_index(self.read_range) self._chunksize = header[8] self._extents = _chunk_extents(self._offsets, header) try: diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 46c2d384b..181f93cf1 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -195,8 +195,9 @@ def test_blocks_are_read_over_ranges(subscriber, any_chunk_wants_blocks): sub.log.clear() assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - # The frame index, then one read for the block offsets and one for the block - assert [kind for kind, _, _ in sub.log] == ["fetch"] * 6 + # The frame index (header, then offsets), one read for the chunk's block + # offsets, and one for the block the slice lands in + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 4 assert {status for _, status, _ in sub.log} == {206} assert not _bytes(sub, "chunk") # A block of a chunk, not the chunk: an eighth of it here, and never the frame diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index b988f0dcb..3d4f2b625 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -759,6 +759,63 @@ def _incompressible(shape, chunks, blocks, seed=0): return data, blosc2.asarray(data, chunks=chunks, blocks=blocks) +def test_lazy_open_costs_two_reads(monkeypatch): + # The format asks four questions to find the chunk offsets; both of the two + # that only measure the next read are guessed at instead + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("openreads.b2nd", a) + reads, chunks = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 2 + assert not chunks + assert src.shape == (600, 600) + assert len(src._offsets) == 2 # ... and it read the offsets it came for + + +def test_lazy_open_of_a_small_frame_costs_one_read(monkeypatch): + # A frame that fits in the first read is wholly in hand: the offsets chunk + # is in those bytes too, so there is nothing left to ask for + a = blosc2.arange(0, 100, dtype="i4", chunks=(10,)) + assert a.schunk.cbytes < blosc2.proxy._FRAME_PREFETCH + url = _put("smallframe.b2nd", a) + reads, _ = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 1 + assert len(src._offsets) == 10 + assert np.array_equal(blosc2.Proxy(src)[:], a[:]) + + +def test_lazy_open_reads_a_header_that_did_not_fit(monkeypatch): + # A metalayer big enough to push the header past the guess: the exact read + # the format asks for happens after all, and nothing is misread + monkeypatch.setattr(blosc2.proxy, "_FRAME_PREFETCH", 256) + data = np.arange(1000, dtype="i4") + a = blosc2.asarray(data, chunks=(100,), meta={"big": {"pad": "x" * 4096}}) + url = _put("bigheader.b2nd", a) + reads, _ = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 3 # the guess, the header, the offsets + assert reads[1] > 4096 + assert np.array_equal(blosc2.Proxy(src)[:], data) + + +def test_lazy_open_reads_an_index_that_did_not_fit(monkeypatch): + # The same for the offsets chunk, which is bounded by the frame's own length + # but capped in case a large trailer sits behind it + monkeypatch.setattr(blosc2.proxy, "_INDEX_PREFETCH", 16) + data, a = _incompressible((600, 600), (30, 600), (30, 600)) + url = _put("bigindex.b2nd", a) + reads, _ = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 3 # the head, the capped tail, the offsets in full + assert reads[1] == 16 + assert np.array_equal(blosc2.Proxy(src)[:], data) + + def test_lazy_fetches_only_touched_blocks(monkeypatch): # Chunks big enough to be worth taking apart, at the real threshold data, a = _incompressible((600, 600), (300, 600), (30, 600)) From b2918c896ca9db52ece1f8d4de149ef4815c0bcf Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:00:26 +0200 Subject: [PATCH 12/25] Notice a remote array that was replaced under a proxy's cache A cache was checked against the source's geometry alone, so a dataset rewritten with the same shape and partitioning was adopted and went on being served from whatever the earlier run had fetched. In block mode that is worse than stale data: the cached chunks were spliced at offsets read from the frame the cache was built against, and the new frame's offsets are somewhere else. C2Array now has a stamp, like FsspecNDSource: api/info's mtime and the compressed size, both already fetched at construction, so it costs no request. The size is in there because a rewrite inside one clock tick is what an mtime cannot see. A subscriber reporting no mtime leaves the array unstamped and the cache checked on geometry alone, as before. The vlmeta entry is renamed fsspec-stamp -> proxy-stamp, since it is no longer only fsspec's. Caches from earlier builds of this development cycle are not adopted; mode="w" once is the way through, and nothing released is affected. Also: do not stamp a cache opened read-only. blosc2.open(path, mode="r") over a persisted proxy hands the Proxy a cache it may not write to, and recording the stamp there raised "Cannot do this action with reading mode" instead of opening it -- reachable only once a C2Array had a stamp to record, which is why the network tests caught it and the local ones did not. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 7 +- plans/cat2-block-granularity.md | 9 ++- src/blosc2/c2array.py | 19 +++++ src/blosc2/proxy.py | 16 ++-- src/blosc2/schunk.py | 2 +- tests/ndarray/test_c2array_blocks.py | 106 +++++++++++++++++++++++++-- tests/ndarray/test_proxy.py | 2 +- 7 files changed, 143 insertions(+), 18 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 79419be93..5bd236902 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -68,7 +68,12 @@ XXX version-specific blurb XXX earlier run instead of failing on the existing file, so a proxy's cache can outlive the process. The cache must come from a proxy over a source of the same shape and dtype; anything else at that path raises. A cache that holds only - some blocks of a chunk keeps them across runs too. + some blocks of a chunk keeps them across runs too. Sources that can name the + bytes they read are held to that as well, so a remote array *replaced* while + keeping its shape is noticed rather than served stale: `FsspecNDSource` uses + fsspec's token and `C2Array` the subscriber's mtime, both free with metadata + they already fetch. Caches from earlier 4.11.1 development builds are not + adopted (the stamp moved to a `proxy-stamp` entry); pass `mode="w"` once. * Querying a `utf8()` column through its FULL index no longer materializes the index vocabulary. The query literal is turned into an alphabetical rank by diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 1c79d2261..90c5bab38 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -415,7 +415,8 @@ in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, an against cat2.cloud. Getting to one for a large frame needs a suffix range (`Range: bytes=-65536`) batched with the head read, which no fsspec backend exposes and `read_ranges` has no way to express. -- **`C2Array` still has no `stamp`.** A cache is checked against the source's - geometry only, so a dataset replaced underneath while keeping its shape is not - noticed. `mtime` is in `api/info` and would do it, but adding one invalidates - every cache built before it, which wants its own decision. +- ~~**`C2Array` still has no `stamp`.**~~ It has one now (*Notice a remote array + that was replaced*): `api/info`'s mtime and the compressed size, so a cache + built from other bytes raises instead of being served stale. The vlmeta entry + is `proxy-stamp` rather than `fsspec-stamp`, since it is no longer only + fsspec's; caches from earlier builds of this cycle are not adopted. diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index d2c1649dc..71a76777d 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -655,6 +655,25 @@ async def aclose(self) -> None: # slice touches instead of whole chunks, wherever that is the cheaper way # round; every one of them falls back to `get_chunk` when it is not. + @property + def stamp(self) -> str | None: + """What names the exact remote bytes, for a :ref:`Proxy` to check a cache by. + + Geometry cannot tell a dataset that was replaced from the one a cache was + filled from: a shape and a partitioning survive a rewrite, while every + cached chunk -- and, in block mode, every offset they were fetched by -- + goes stale. The subscriber's own mtime does tell, and `api/info` carries + it, so this costs no request; the compressed size goes in with it, since + a rewrite within the same clock tick is what an mtime cannot see. + + None when the subscriber reports no mtime, which leaves the cache checked + on its geometry alone, as every source without a stamp is. + """ + mtime = self.meta.get("mtime") + if mtime is None: + return None + return f"{mtime}:{self.meta['schunk'].get('cbytes', '')}" + @property def blocks_per_chunk(self) -> int: """How many blocks a chunk of the remote array holds. diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index a323acd8e..ce75dd869 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -72,7 +72,7 @@ # vlmeta entries the proxy keeps its own state in: what it has fetched, and which # remote bytes the cache was filled from. A caller cannot write these. -_RESERVED_VLMETA = frozenset({"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "fsspec-stamp"}) +_RESERVED_VLMETA = frozenset({"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "proxy-stamp"}) # `jit` kwargs that tune *how* an expression is evaluated, not what container the # result is stored in. Unlike storage kwargs (`cparams`, `chunks`, `urlpath`, ...), @@ -294,7 +294,8 @@ def __init__( else raises rather than being silently reused or overwritten. A source that can name the exact bytes it reads, as - :ref:`FsspecNDSource` does with its ``stamp``, is checked against that + :ref:`FsspecNDSource` does with its ``stamp`` (fsspec's token) and + :ref:`C2Array` with the subscriber's mtime, is checked against that too: a cache built from different bytes raises, even when the geometry still fits. For every other source geometry is all there is to check, so a source whose contents changed underneath while its geometry did @@ -396,14 +397,17 @@ def __init__( # Geometry alone cannot tell a replaced source from the one the cache was # filled from, so record whatever identity the source can name itself by stamp = getattr(self.src, "stamp", None) - if stamp is not None: - self._schunk_cache.vlmeta["fsspec-stamp"] = stamp + if stamp is not None and getattr(self._schunk_cache, "mode", None) != "r": + # Not into a cache opened read-only, which `blosc2.open(path, mode="r")` + # hands over for a persisted proxy: nothing may be written there, and a + # proxy over one stays observational anyway (see `__getitem__`) + self._schunk_cache.vlmeta["proxy-stamp"] = stamp if vlmeta: reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) if reserved: # Writing these would hand the proxy a bitmap or an identity it # never earned: a caller's `proxy-fetched` makes it skip chunks it - # has not fetched, and a caller's `fsspec-stamp` makes a good cache + # has not fetched, and a caller's `proxy-stamp` makes a good cache # fail its identity check (or a stale one pass it) raise ValueError( f"{', '.join(reserved)} {'is' if len(reserved) == 1 else 'are'} reserved " @@ -594,7 +598,7 @@ def _reopen_cache(self, urlpath: str): # goes stale. Only for sources that can name themselves; the rest are # adopted on geometry alone, as documented. stamp = getattr(self.src, "stamp", None) - if stamp is not None and schunk.vlmeta.get("fsspec-stamp") != stamp: + if stamp is not None and schunk.vlmeta.get("proxy-stamp") != stamp: raise ValueError( f"the cache at {urlpath} was built against different remote bytes; " f"pass mode='w' to fetch them anew" diff --git a/src/blosc2/schunk.py b/src/blosc2/schunk.py index f6e32e90a..c06ba2b9f 100644 --- a/src/blosc2/schunk.py +++ b/src/blosc2/schunk.py @@ -2011,7 +2011,7 @@ def _cache_stamp(path: str): cache = blosc2_ext.open(path, "r", 0, **kwargs) except RuntimeError: return None - return getattr(cache, "schunk", cache).vlmeta.get("fsspec-stamp") + return getattr(cache, "schunk", cache).vlmeta.get("proxy-stamp") def _open_fsspec_url(urlpath: str, mode: str, offset: int, kwargs: dict): diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 181f93cf1..9376144d3 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -18,6 +18,7 @@ import contextlib import json import math +import os import pathlib import threading from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer @@ -33,13 +34,18 @@ class _Subscriber: def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges=True): self.path = str(path) - self.frame = pathlib.Path(self.path).read_bytes() - self.array = blosc2.open(self.path) self.ranges = ranges # False: stream the body and ignore Range, as a self.cookie = cookie # computed dataset does self.multipart = multipart # False: answer only the first range asked for self.merge_ranges = merge_ranges # as Starlette does with ranges that touch self.log = [] # (endpoint, status, bytes served) + self.reload() + + def reload(self): + """Pick up the file as it is now, as a subscriber would on the next request.""" + self.frame = pathlib.Path(self.path).read_bytes() + self.array = blosc2.open(self.path) + self.mtime = pathlib.Path(self.path).stat().st_mtime @property def meta(self): @@ -49,7 +55,7 @@ def meta(self): "chunks": list(self.array.chunks), "blocks": list(self.array.blocks), "dtype": str(self.array.dtype), - "mtime": None, + "mtime": self.mtime, "schunk": { "cparams": {"typesize": self.array.dtype.itemsize}, "nbytes": schunk.nbytes, @@ -329,9 +335,10 @@ def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_bl array._block_source = None # as if the subscriber served no ranges p = blosc2.Proxy(array, urlpath=cache, mode="a") assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - del p, array + del p - array, _ = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + # The same dataset, opened afresh: this one takes the blocks path + array = blosc2.C2Array(array.path, urlbase=array.urlbase) p = blosc2.Proxy(array, urlpath=cache, mode="a") served = len(sub.log) assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) @@ -407,3 +414,92 @@ def test_a_server_that_answers_one_range_stops_being_batched(subscriber, any_chu # One request per range from here on, and no second attempt at batching assert len(sub.log) > served + 2 assert np.array_equal(p[...], data) + + +# --- a cache is checked against the bytes it was filled from ---------------- + + +def _replace(sub, data, chunks, blocks): + """Rewrite the served dataset, as an upload of new data would.""" + blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=sub.path, mode="w") + stat = pathlib.Path(sub.path).stat() + os.utime(sub.path, (stat.st_atime, stat.st_mtime + 10)) # a tick the clock cannot swallow + sub.reload() + + +def test_a_cache_is_stamped_with_the_remote_mtime(subscriber): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + + assert array.stamp == f"{sub.mtime}:{sub.array.schunk.cbytes}" + assert p.schunk.vlmeta["proxy-stamp"] == array.stamp + + +def test_a_cache_from_other_bytes_is_refused(tmp_path, subscriber, any_chunk_wants_blocks): + # Same shape, same partitioning, different data: geometry cannot tell, and + # every cached chunk (and the offsets it was fetched by) is stale + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "stamped.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + other = _incompressible((200, 200), seed=1) + _replace(sub, other, chunks=(100, 200), blocks=(10, 20)) + replaced = blosc2.C2Array(array.path, urlbase=array.urlbase) + assert replaced.stamp != array.stamp + + with pytest.raises(ValueError, match="different remote bytes"): + blosc2.Proxy(replaced, urlpath=cache, mode="a") + + # ... and mode="w" is the way through, with the new data behind it + p = blosc2.Proxy(replaced, urlpath=cache, mode="w") + assert np.array_equal(p[0:5, 0:10], other[0:5, 0:10]) + assert np.array_equal(p[...], other) + + +def test_a_cache_from_the_same_bytes_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "unchanged.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + assert again.stamp == array.stamp + p = blosc2.Proxy(again, urlpath=cache, mode="a") + served = len(sub.log) + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert len(sub.log) == served # what it holds was not fetched again + + +def test_no_stamp_when_the_subscriber_reports_no_mtime(tmp_path, subscriber): + # Then the cache is checked on geometry alone, as every unstamped source is + data = _incompressible((200, 200)) + array, _sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + del array.meta["mtime"] + assert array.stamp is None + + cache = str(tmp_path / "unstamped.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert "proxy-stamp" not in p.schunk.vlmeta.getall() + del p + assert blosc2.Proxy(array, urlpath=cache, mode="a") is not None + + +def test_a_read_only_cache_is_not_stamped(tmp_path, subscriber): + # `blosc2.open(path, mode="r")` rebuilds the proxy over a cache that may not + # be written to; recording the stamp there raised instead of opening it + data = _incompressible((200, 200)) + array, _sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "readonly.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="w") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + reopened = blosc2.open(cache, mode="r") + assert np.array_equal(reopened[0:5, 0:10], data[0:5, 0:10]) diff --git a/tests/ndarray/test_proxy.py b/tests/ndarray/test_proxy.py index 1fffce5aa..3b611a84d 100644 --- a/tests/ndarray/test_proxy.py +++ b/tests/ndarray/test_proxy.py @@ -369,7 +369,7 @@ def test_evicted_chunk_is_fetched_again(): def test_vlmeta_cannot_overwrite_proxy_state(): # A caller-supplied bitmap would make the proxy skip chunks it never fetched source = blosc2.asarray(np.arange(20).reshape(4, 5), chunks=(2, 5), blocks=(1, 5)) - for name in ("proxy-fetched", "proxy-fetched-blocks", "fsspec-stamp"): + for name in ("proxy-fetched", "proxy-fetched-blocks", "proxy-stamp"): with pytest.raises(ValueError, match="reserved"): blosc2.Proxy(source, vlmeta={name: b"nonsense"}) # Anything else still goes through From 7b9aa6230160c2dbe69503fc252dc98fb17671ac Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:03:25 +0200 Subject: [PATCH 13/25] Record the whole of the block-granularity work in the plan The four follow-ons (the bench, the container-leaf chunks, the two-request open, the stamp) were only visible as strikethroughs in a list of things left undone, and the headline numbers predated the last two commits. Both are now one run of the bench against cat2.cloud, which is where those numbers should come from anyway. Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 153 ++++++++++++++++++++------------ 1 file changed, 95 insertions(+), 58 deletions(-) diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 90c5bab38..13d62df53 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -3,10 +3,12 @@ Written 2026-08-17, after [plans/fsspec-blocks.md](fsspec-blocks.md) landed block fetching for fsspec URLs (merged as PR #701). -**All five phases are implemented** (2026-08-17). Phases 1-4 are in blosc2 on -`cat2-block-granularity`; phase 5 is in Caterva2 on `range-honesty`. See -[what landed](#what-landed) at the end for the results and the two things the -work found out. +**All five phases are implemented**, and four more things that came out of them +(2026-08-17/18). The blosc2 side is on `cat2-block-granularity`, the Caterva2 +side on `range-honesty`. Everything from *What was verified* to *Reproducing the +measurements* is the analysis as it stood before any of it was written; see +[what landed](#what-landed) at the end for what was built, what it measures at, +and where the analysis turned out to be wrong. ## The question @@ -329,8 +331,11 @@ question for an fsspec URL. ## What landed -Every phase, in the order the plan gives them. Where the plan guessed and the -work found out otherwise, that is said below rather than quietly fixed. +All five phases, and four more things the work turned up on the way. Where the +plan guessed and the work found out otherwise, that is said below rather than +quietly fixed. + +### The five phases | phase | where | commit | |---|---|---| @@ -338,85 +343,117 @@ work found out otherwise, that is said below rather than quietly fixed. | 2. capability check | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | | 3b. `ByteRangeNDSource` | blosc2 `proxy.py` | *Lift the frame reading out of FsspecNDSource* | | 3. `C2Array` blocks | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | -| 4. multipart | blosc2 both | *Ask a subscriber for a whole wave of ranges at once* | -| 5. honest streaming | caterva2 `server.py` | *Say which responses serve byte ranges* | +| 4. multipart | blosc2 `proxy.py`, `c2array.py` | *Ask a Caterva2 subscriber for a whole wave of ranges at once* | +| 5. honest streaming | caterva2 `server.py` | *Say which responses serve byte ranges, and refuse the rest* | Option **(b)** was taken for phase 3, as recommended: `ByteRangeNDSource` holds the frame format and one abstract `read_range`, `FsspecNDSource` is that plus four lines of fsspec, and `C2NDSource` is that plus HTTP ranges with the auth -cookie. `C2Array` keeps the five members `Proxy` looks for and delegates them, +cookie. `C2Array` keeps the five members `Proxy` looks for and delegates them, so every existing `Proxy(C2Array(...))` gets blocks without being asked. +### And four things that came out of it + +| | where | commit | +|---|---|---| +| a bench for all of it | blosc2 `bench/ndarray/` | *Measure the Caterva2 block path the way the fsspec one is measured* | +| chunks of a container leaf | caterva2 `server.py` | *Serve chunks of a container leaf, and refuse the ones that are not stored* | +| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four* | +| a stamp for `C2Array` | blosc2 `c2array.py` | *Notice a remote array that was replaced under a proxy's cache* | + +- **`bench/ndarray/cat2-block-granularity.py`** is the fsspec bench's question + asked of a subscriber, and it needs no service to point at: a stand-in serves + `api/info`, `api/fetch` and `api/chunk` over loopback from any local `.b2nd`, + ranges, multipart and Starlette's sort-and-merge included. It reports whether + a dataset serves ranges at all and what finding out cost, the request plan of + each mode, what a pooled connection is worth, and the timed slices. +- **`api/chunk` serves container leaves**, which it never did: it was the last + read endpoint resolving with `get_abspath` rather than `split_and_resolve`, so + a `Proxy` over a `.b2z` leaf could not fetch anything at all. HDF5 leaves and + CTables are refused with a 400 naming `slice_` instead of being recompressed + per request, and the opened leaf is cached (keyed on the container's mtime, as + `get_filtered_array` is) so a chunk-by-chunk read does not reopen the `.b2z` + every time. +- **Opening a frame costs two requests**, or one when the frame arrives whole in + the first read: the two reads that only measured the next one are guessed at + instead (8 KB of head, and a tail the frame's own length bounds). It is in + `ByteRangeNDSource`, so `blosc2.open(url, lazy=True)` gets it too. +- **`C2Array` has a stamp**: `api/info`'s mtime and the compressed size, so a + cache built against a dataset that was since replaced raises instead of being + served stale. In block mode that is worse than stale data -- the cached chunks + were spliced at offsets read from the frame the cache was built against. + ### The numbers, end to end -Against `cat2.cloud/demo` on `kevlar-tomo.b2nd` (1.44 MB chunks, 47 blocks each): +`bench/ndarray/cat2-block-granularity.py @public/examples/kevlar-tomo.b2nd +--urlbase https://cat2.cloud/demo` (1.44 MB chunks, 47 blocks each), median of 5: -| | requests | bytes | time | -|---|---|---|---| -| chunks (before) | 2 | 2.723 MB | 0.28 s | -| blocks | 8 | 0.031 MB | 0.34 s | +| pattern | chunk mode | blocks | multipart | vs chunks | +|---|---|---|---|---| +| point | 1 req, 1.38 MB, 0.169 s | 2 req, 0.02 MB, 0.097 s | 2 req, 0.02 MB, 0.088 s | 1.9x | +| line, last dim | 1 req, 1.38 MB, 0.140 s | 2 req, 0.02 MB, 0.087 s | 2 req, 0.02 MB, 0.090 s | 1.6x | +| line, first dim | 10 req, 14.43 MB, 0.956 s | 20 req, 0.20 MB, 0.198 s | **2 req, 0.20 MB, 0.111 s** | **8.6x** | +| window (1/64 per dim) | 1 req, 1.38 MB, 0.160 s | 2 req, 0.02 MB, 0.092 s | 2 req, 0.02 MB, 0.089 s | 1.8x | +| slab (1% of dim 0) | 1 req, 1.38 MB, 0.169 s | 1 req, 1.38 MB, 0.155 s | 1 req, 1.38 MB, 0.161 s | 1.0x | +| slab (10% of dim 0) | 1 req, 1.38 MB, 0.155 s | 1 req, 1.38 MB, 0.148 s | 1 req, 1.38 MB, 0.149 s | 1.0x | -88x fewer bytes for a corner slice, and the same wall time on a link where a -round trip and a megabyte cost about the same. For a slice touching ten chunks, -where the request count is what decides: +1.4% of the bytes for anything that lands in a corner of a chunk, and the two +slabs that want every block of theirs come out identical in all three modes, +which is the threshold declining to take a chunk apart. Around it: -| | requests | time | +| | before | after | |---|---|---| -| blocks, one request at a time | 20 | 1.008 s | -| blocks, fetches overlapped | 20 | 0.334 s | -| blocks, multipart | 2 | 0.141 s | +| one request (`api/info`) | 163.3 ms, a client per request | **42.3 ms**, pooled (3.9x) | +| opening a `C2Array` | 0.237 s, 4 requests, 303 bytes | **0.138 s**, 2 requests, 8306 bytes | +| a wave of 32 ranges | 1.530 s one at a time, 0.208 s eight at a time | **0.136 s**, one multipart request | -Phase 1 on its own: 0.162 s per request against 0.046 s pooled, on the existing -chunk path. +Phase 4 was built because that last row said to. Starlette *sorts and merges* +the spans it is given and answers a plain 206 when they all merge into one, so +the client maps the parts back by what each says it holds rather than by order; +a server that answers with less than was asked for is noticed once and never +batched again. -### Two things the plan had wrong +### What the work found out - **`C2Array` cannot be built over a computed dataset at all.** `api/info` for a lazy expression carries no `schunk`, and `C2Array.__init__` reads - `meta["schunk"]["cparams"]`, so it raises long before any of this. The + `meta["schunk"]["cparams"]`, so it raises long before any of this. The info-based discriminator of phase 2 is still there and still right, but it earns its place on the *other* case: - **A `.b2z` member reports a full geometry and is streamed.** Confirmed against a local server: `api/info` on `@public/tree-store.b2z/level1/leaf6` answers - with `blocks`, `chunks` and `schunk`, and `api/fetch` streams it. So the + with `blocks`, `chunks` and `schunk`, and `api/fetch` streams it. So the status code of the first range read is the authority, exactly as phase 2 - argued — with phase 5 in place that costs 169 bytes and one round trip. - -Phase 4 was built because the measurement said to: 32 spans cost 0.136 s in one -multipart request against 0.208 s as 32 requests eight at a time, and 1.530 s -one at a time. Starlette *sorts and merges* the spans it is given and answers a -plain 206 when they all merge into one, so the client maps parts back by what -each says it holds rather than by order; a server that answers with less than -was asked for is noticed once and never batched again. - -The plan's guess that `_run` would already overlap a C2Array's fetches was wrong -in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, and -`C2Array` had none, so the sync path was serial. It has one now, the same 8 -`afetch` already used. + argued -- with phase 5 in place that costs 169 bytes and one round trip. +- **A `Proxy` over a container leaf never worked**, which is how the bench found + the `api/chunk` gap: it died in httpx rather than reporting anything. Fixed + above; the bench now says so plainly when a dataset has no chunks to serve. +- **`Proxy.fetch` was serial over a C2Array.** It reads `max_concurrency` off the + source and `C2Array` had none, so the plan's assumption that a chunk's runs + already go out in parallel was wrong. It has one now, the same 8 `afetch` + already used, and that alone takes the unbatched block path from 1.008 s to + 0.334 s. +- **A read-only cache cannot be stamped.** `blosc2.open(path, mode="r")` over a + persisted proxy hands `Proxy` a cache it may not write to, and recording the + stamp there raised instead of opening it -- reachable only once a `C2Array` had + a stamp to record, and caught by the network tests rather than the local ones. ### Left undone -- ~~**`api/chunk` does not serve container members.**~~ Fixed in Caterva2 on - `range-honesty` (*Serve chunks of a container leaf*): the endpoint resolves the - way `api/fetch` does, so a TreeStore leaf hands over its stored chunk, while - HDF5 leaves and CTables are refused with a 400 naming `slice_` rather than - being recompressed per request. A `.b2z` leaf still gets whole chunks only: - giving one the block path needs the offset of its frame inside the container, - which is the next item. - **A container leaf could serve ranges too.** A TreeStore keeps its leaves as ordinary frames inside the `.b2z`, so the bytes a block reader wants are in the file at a fixed offset -- what is missing is a way for the server to say where a leaf's frame starts, and for the client to add that base to every - range. Worth its own plan; it would give `.b2z` members everything a plain + range. Worth its own plan; it would give `.b2z` members everything a plain `.b2nd` has. -- ~~**The four requests to open a frame**~~ are two, and one for a frame that - arrives whole in the first read (*Open a frame in two requests*): both reads - that only measured the next one are guessed at instead. 0.237 s → 0.138 s - against cat2.cloud. Getting to one for a large frame needs a suffix range - (`Range: bytes=-65536`) batched with the head read, which no fsspec backend - exposes and `read_ranges` has no way to express. -- ~~**`C2Array` still has no `stamp`.**~~ It has one now (*Notice a remote array - that was replaced*): `api/info`'s mtime and the compressed size, so a cache - built from other bytes raises instead of being served stale. The vlmeta entry - is `proxy-stamp` rather than `fsspec-stamp`, since it is no longer only - fsspec's; caches from earlier builds of this cycle are not adopted. +- **Opening a frame could be one request rather than two**, with a suffix range + (`Range: bytes=-65536`) batched with the head read. No fsspec backend exposes + suffix ranges and `read_ranges` has no way to express one, so it would be a + C2-only path through the shared base -- which is why it was not taken. +- **Peer datasets stay on the streaming side.** They are fetched from their owner + and re-serialized here, so there is no file to seek into; phase 5 makes them + say so, and that is all they can do without a design of their own. +- **Quota and accounting**, from the risks above, is still undecided: block reads + replace one chunk GET with a handful of small ranged ones, and multipart puts + a whole wave in one. A subscriber metering per request would see block clients + as expensive while they transfer far less. From b6b7303a26822c4857090aaa4e33fa6d9a775947 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:09:41 +0200 Subject: [PATCH 14/25] Plan byte ranges for container leaves A .b2z is a zip of stored frames, so a leaf's bytes are a contiguous frame at a fixed offset and blosc2 already computes where. If api/fetch answers a leaf's ranged request by seeking into the container, every client that reads a .b2nd over ranges reads a leaf the same way, knowing nothing about containers. Co-Authored-By: Claude Opus 5 --- plans/container-leaf-ranges.md | 171 +++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 plans/container-leaf-ranges.md diff --git a/plans/container-leaf-ranges.md b/plans/container-leaf-ranges.md new file mode 100644 index 000000000..8e1ab4d16 --- /dev/null +++ b/plans/container-leaf-ranges.md @@ -0,0 +1,171 @@ +# Byte Ranges For Container Leaves + +Analysis and plan, written 2026-08-18, after +[plans/cat2-block-granularity.md](cat2-block-granularity.md) gave `C2Array` the +block path and left this as the one thing a `.b2z` leaf still could not have. + +## The question + +A `Proxy` over `@public/tree.b2z/g/a` fetches whole chunks. A proxy over +`@public/a.b2nd` fetches the blocks a slice touches, because Caterva2 serves a +stored dataset from its file and `Range` works on it. A leaf is stored too -- +inside a container -- so what stands between it and the same treatment? + +**Verdict: one server change, and no client change at all.** A leaf's frame is a +contiguous, self-consistent Blosc2 frame at a fixed offset in the `.b2z`, and +blosc2 already knows where. If `api/fetch` answers a leaf's ranged request by +seeking into the container, every client that can read a `.b2nd` over ranges +reads a leaf the same way, knowing nothing about containers. + +## What was verified + +Against a local server from `~/ironArray/caterva2` and stores written by +`blosc2.TreeStore`. + +### A `.b2z` is a zip of *stored* frames + +``` +a.b2nd method=0 size=287667 header_offset=0 +g/b.b2nd method=0 size= 1835 header_offset=287703 +embed.b2e method=0 size= 245 header_offset=289576 +``` + +`method=0` is the whole point: every member is written uncompressed, so a +member's bytes in the file *are* the frame. Each one is self-consistent -- +`b2frame` magic at byte 2, and the `frame_len` in its header equals the member's +length -- so a reader handed the window sees an ordinary frame beginning at 0. + +blosc2 computes the windows already, for its own reading: `DictStore.offsets` +maps a zip member to `{offset, length}` (parsed off the local file headers) and +`DictStore.map_tree` maps a logical key to that member: + +``` +map_tree['/g/deep'] = 'g/deep.b2nd' +offsets['g/deep.b2nd'] = {'offset': 2901, 'length': 240} +``` + +Every leaf of a `TreeStore` is such a member, whether it was given an `NDArray`, +a plain NumPy array or an `SChunk` (`.b2nd`, `.b2f`). + +**The stored member and `leaf.to_cframe()` are not the same bytes.** Same +length, same data, same vlmeta, but not byte-identical -- what `api/fetch` +serves today for a whole member is a re-serialization, not the file. Nothing +depends on the two agreeing (a client reads geometry from `api/info` and the +frame index over ranges), but it decides one thing in the plan below: the +whole-member path is left alone. + +### An embedded leaf is not addressable, and that is right + +`EmbedStore` keeps its members inside a Blosc2 super-chunk of its own +(`self._store[offset:offset+len] = serialized`), which is compressed, so an +embedded leaf has no raw window in the file. A `C2Array` leaf is a reference +rather than bytes at all. Both must say "no window" rather than a wrong one -- +the same rule as before: **blocks where the bytes already exist as a frame**. + +### The server has no route to those bytes today + +``` +GET api/fetch/@public/inspect.b2z -> 500 (!) +GET api/fetch/@public/inspect.b2z/a Range: -> 416 (correct today: it is built, not stored) +GET api/download/@public/inspect.b2z Range: -> 416 (correct: download never serves ranges) +``` + +The 500 is a bug of its own: a `TreeStore` container falls through +`fetch_data`'s type ladder into the `SChunk` branch and dies on +`schunk.typesize`, where the docstring promises "its stored image is served". + +### What the client would need: nothing + +`C2Array` over a leaf already gets its geometry from `api/info` (a leaf reports +`chunks`, `blocks` and `schunk` like any array), already probes with a `Range`, +already falls back to `api/chunk` when the probe fails, and already reads a +frame that starts at 0. If the server maps a leaf's ranges onto the container, +all of that works unchanged. + +## What not to build + +- **A new endpoint.** `api/fetch` is where a dataset's bytes come from, leaf or + not; a second door would need its own auth, its own tests and its own client. +- **A client that parses the zip itself.** It would have to know the zip layout + *and* blosc2's member-naming convention, and there is no route to the + container's bytes anyway (`api/fetch` on a container 500s, `api/download` + refuses ranges) -- so it needs a server change regardless, and a bigger one. +- **Serving a whole member from its window.** Cheaper than rebuilding the + cframe, and tempting, but it would change the bytes clients get today for the + sake of an optimization nobody asked for. +- **HDF5 leaves.** An HDF5 dataset is not a Blosc2 frame; there is no window to + hand over. They keep the 400 that names `slice_`. + +## Plan + +### Phase 1 — `DictStore.member_window(key)` (blosc2) + +The format knowledge belongs where the format is: + +```python +def member_window(self, key: str) -> tuple[int, int] | None: + """Where the frame behind *key* lies in the ``.b2z``, as (offset, nbytes).""" +``` + +`None` for a directory-backed store, an embedded leaf, a `C2Array` reference, or +a key that names no leaf. Built out of `map_tree` and `offsets`, which are +already maintained; ~10 lines and a test that the window decodes to the leaf. + +### Phase 2 — Serve a leaf's ranges from the container (caterva2) + +In `fetch_data`, where a member currently refuses every range: + +- look the window up (through the cached container opened for `api/chunk`); +- with no `Range`, answer as today (`Accept-Ranges: bytes` instead of `none`); +- with one, answer 206 from the container file, offset by the window's start and + clamped to its length -- a range must never reach past the leaf it names, both + because the client's frame would make no sense and because the window is the + only part of the container this path is about. + +RFC 7233 by hand, which is ~60 lines: parse, sort, merge what touches, single +206 with `Content-Range` or `multipart/byteranges` for several, 416 with +`Content-Range: bytes */len` for the unsatisfiable, 400 for the malformed. The +client already survives merged and reordered parts, since Starlette does that to +ranges too, and falls back to a range per request against a server that answers +with less than it asked for -- so multipart is worth implementing rather than +leaving to that fallback. + +### Phase 3 — A container fetched whole stops being a 500 (caterva2) + +`api/fetch/{container}` should return the stored image, as its docstring says: +the `.b2z` itself, through the same `FileResponse` a `.b2nd` gets. Two lines and +a test. Independent of the rest, and worth doing on its own. + +### Phase 4 — Tests, and the bench + +Server: a leaf's ranged request returns exactly the container's bytes at the +window; a range past the leaf's end is clamped; a multi-range request comes back +multipart; an embedded leaf and an HDF5 leaf still refuse; the whole-member fetch +still returns what it always did. + +Client: nothing changes, which is the thing to assert -- a `Proxy` over a leaf +reads blocks, over the stand-in subscriber, and its traffic is a fraction of the +chunk path's. + +Then `bench/ndarray/cat2-block-granularity.py @public/tree.b2z/leaf` against a +local server, which should print `byte ranges: served` where it printed `not +served` and time the three modes as it does for a plain array. + +## Risks and open questions + +- **A hand-rolled RFC 7233.** Starlette's is a classmethod on `FileResponse` and + could be reused, at the price of depending on a private API; the alternative + is ~25 lines of parsing with tests of its own. Either way the client is the + same one that already reads Starlette's answers, so the two must agree on + merging and on what a 416 looks like. +- **The window is only as fresh as the store index.** The opened container is + cached keyed on its mtime (as `api/chunk` does), so a rewritten `.b2z` gets + new windows; a `.b2z` rewritten *in place* within one mtime tick would serve + stale offsets, which is the same exposure every cached read here has. +- **A leaf has no mtime of its own**, so a `C2Array` over one stamps its proxy + cache with the container's. Coarse but never wrong: rewriting any leaf + rewrites the container. +- **Compressed zip members would break this silently.** Nothing in blosc2 writes + them today (`method=0` throughout), but a `.b2z` produced by another tool + could, and its window would decode to nonsense. The member's own frame magic + is what catches that, on the server, before the window is offered. From e0cc7da670ebaaee8ed06e0029a0247ad6a8a1d0 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:11:36 +0200 Subject: [PATCH 15/25] Say where a leaf's frame lies inside a .b2z A zip store writes each external leaf uncompressed, so a member's bytes in the file are the Blosc2 frame that leaf would have been written as on its own: self-contained, beginning at a known offset, and readable by anything that reads a frame. The store computes those windows already, for its own reading -- map_tree names the member, offsets says where it is -- but nothing could ask. member_window(key) is that question, and it answers None where there is no window rather than a wrong one: a directory-backed store keeps leaves in files of their own, an embedded leaf lives inside the store's own compressed super-chunk, and a C2Array leaf is a reference rather than bytes. Wanted by a server that would hand a byte-range reader the window instead of rebuilding the leaf per request, which is plans/container-leaf-ranges.md. Co-Authored-By: Claude Opus 5 --- doc/reference/dict_store.rst | 6 ++++- src/blosc2/dict_store.py | 45 ++++++++++++++++++++++++++++++++++++ tests/test_dict_store.py | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 92 insertions(+), 1 deletion(-) diff --git a/doc/reference/dict_store.rst b/doc/reference/dict_store.rst index 38bd00579..fd3078af6 100644 --- a/doc/reference/dict_store.rst +++ b/doc/reference/dict_store.rst @@ -90,7 +90,11 @@ Quick example ----------- Use :meth:`DictStore.to_b2z` to pack a directory-backed store into a ``.b2z`` archive, and :meth:`DictStore.to_b2d` to materialize a store as a - ``.b2d`` directory. + ``.b2d`` directory. :meth:`DictStore.member_window` says where a leaf's + frame lies inside a ``.b2z``, for a reader that can take a byte window + rather than the leaf itself. + + .. automethod:: member_window Public Members -------------- diff --git a/src/blosc2/dict_store.py b/src/blosc2/dict_store.py index 82ada0d3a..035be5971 100644 --- a/src/blosc2/dict_store.py +++ b/src/blosc2/dict_store.py @@ -439,6 +439,51 @@ def _update_map_tree_from_offsets(self): if os.path.splitext(filepath)[1] in external_exts or self._probe_external_leaf_offset(filepath): self.map_tree[self._logical_key_from_relpath(filepath)] = filepath + def member_window(self, key: str) -> tuple[int, int] | None: + """Where the frame behind *key* lies in the ``.b2z``, as ``(offset, nbytes)``. + + A zip store keeps each external leaf as a *stored* member, so those bytes + are the leaf's Blosc2 frame as it would have been written on its own: + self-contained, beginning at ``offset``, and readable by anything that + reads a frame -- ``blosc2.open(path, offset=offset)`` is what this store + does with it, and a server can hand the same window to a byte-range + reader instead of rebuilding the leaf per request. + + None when there is no such window, which is not an error but an answer: + a directory-backed store keeps its leaves in files of their own, an + embedded leaf lives inside the store's own compressed super-chunk, a + :ref:`C2Array` leaf is a reference rather than bytes, and a key that + names a group or nothing at all has none either. + + Parameters + ---------- + key: str + The logical key of the leaf, as :meth:`keys` reports it. + + Returns + ------- + out: tuple or None + ``(offset, nbytes)`` into the ``.b2z`` file, or None. + + Examples + -------- + >>> import numpy as np, blosc2 + >>> with blosc2.TreeStore("win.b2z", mode="w") as tstore: + ... tstore["/a"] = np.arange(1000, dtype="i4") + >>> tstore = blosc2.open("win.b2z") + >>> offset, nbytes = tstore.member_window("/a") + >>> frame = open("win.b2z", "rb").read()[offset : offset + nbytes] + >>> np.array_equal(blosc2.ndarray_from_cframe(frame)[:], np.arange(1000, dtype="i4")) + True + """ + if not self.is_zip_store: + return None + relpath = self.map_tree.get(key) + if relpath is None: + return None + window = self.offsets.get(relpath) + return (window["offset"], window["length"]) if window else None + def _annotate_external_value( self, key: str, diff --git a/tests/test_dict_store.py b/tests/test_dict_store.py index 92bd8bb15..7405c5dd5 100644 --- a/tests/test_dict_store.py +++ b/tests/test_dict_store.py @@ -6,6 +6,7 @@ ####################################################################### import os +import pathlib import shutil import subprocess import sys @@ -852,3 +853,44 @@ def test_dict_store_overwrite_key_across_tiers(tmp_path): dstore["/k"] = np.array([9], dtype=np.int8) # embedded overwrite assert len(dstore) == 1 assert dstore["/k"][:].tolist() == [9] + + +def test_member_window(populated_dict_store): + """Where a leaf's frame lies in a .b2z, for a reader that can take a window. + + A zip store writes each external leaf uncompressed, so those bytes are the + frame the leaf would have been written as on its own -- which is what lets a + server hand a byte-range reader the window instead of rebuilding the leaf. + """ + dstore, path = populated_dict_store + dstore.close() + reopened = blosc2.open(path) + + if path.endswith(".b2d"): # a directory store keeps each leaf in a file + assert all(reopened.member_window(key) is None for key in reopened) + return + + raw = pathlib.Path(path).read_bytes() + windows = {key: reopened.member_window(key) for key in reopened} + assert any(window is not None for window in windows.values()) + for key, window in windows.items(): + if window is None: + # An embedded leaf lives inside the store's own compressed + # super-chunk, so it has no window in the file to point at + assert key in reopened.estore + continue + offset, nbytes = window + frame = raw[offset : offset + nbytes] + assert frame[2:9] == b"b2frame" # a whole frame, starting where it says + assert np.array_equal(blosc2.from_cframe(frame)[:], reopened[key][:]) + # ... and nothing is claimed for what has no window of its own + assert reopened.member_window("/nope") is None + + +def test_member_window_of_a_tree_group(tmp_path): + path = str(tmp_path / "grouped.b2z") + with blosc2.TreeStore(path, mode="w") as tstore: + tstore["/g/leaf"] = np.arange(10, dtype="i4") + tstore = blosc2.open(path) + assert tstore.member_window("/g/leaf") is not None + assert tstore.member_window("/g") is None # a group is not a leaf From 460b060f2b3e3ee6031026c90b4659c90c7d621d Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:20:45 +0200 Subject: [PATCH 16/25] Pin that a container leaf reads its blocks like any other array The client side of container-leaf ranges is that there is no client side: a subscriber serves a leaf out of its window in the .b2z, so what arrives is a frame beginning at 0 and everything already written for a .b2nd applies. The stand-in subscriber can now be pointed at a leaf, which is the same six lines: it serves the window instead of the file, exactly as Caterva2 does. The tests then assert what matters -- a Proxy over a leaf fetches blocks rather than chunks, and the leaf is stamped with the container's mtime, since a leaf has none of its own. Co-Authored-By: Claude Opus 5 --- tests/ndarray/test_c2array_blocks.py | 62 ++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 9376144d3..1d9b3c1bf 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -32,8 +32,9 @@ class _Subscriber: """A Caterva2-shaped server over one .b2nd file.""" - def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges=True): + def __init__(self, path, key=None, ranges=True, cookie=None, multipart=True, merge_ranges=True): self.path = str(path) + self.key = key # a leaf inside a .b2z container, rather than a file of its own self.ranges = ranges # False: stream the body and ignore Range, as a self.cookie = cookie # computed dataset does self.multipart = multipart # False: answer only the first range asked for @@ -42,10 +43,20 @@ def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges= self.reload() def reload(self): - """Pick up the file as it is now, as a subscriber would on the next request.""" - self.frame = pathlib.Path(self.path).read_bytes() - self.array = blosc2.open(self.path) + """Pick up the file as it is now, as a subscriber would on the next request. + + A leaf is served out of its window in the container, which is what makes + it look to a client exactly like a dataset of its own: byte 0 of what it + asks for is the frame's first byte. + """ + raw = pathlib.Path(self.path).read_bytes() self.mtime = pathlib.Path(self.path).stat().st_mtime + if self.key is None: + self.frame, self.array = raw, blosc2.open(self.path) + return + store = blosc2.open(self.path) + offset, nbytes = store.member_window(self.key) + self.frame, self.array = raw[offset : offset + nbytes], store[self.key] @property def meta(self): @@ -152,15 +163,25 @@ def _fetch(self, sub): ) -def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", **kwargs): - """A C2Array over *data*, served by a subscriber stand-in on localhost.""" +def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", key=None, **kwargs): + """A C2Array over *data*, served by a subscriber stand-in on localhost. + + With *key*, the array is a leaf of a TreeStore container instead of a file + of its own, and the subscriber serves it from its window -- which is what + Caterva2 does, and what the client is meant not to notice. + """ urlpath = str(tmp_path / name) - blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") + if key is None: + blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") + else: + with blosc2.TreeStore(urlpath, mode="w") as tstore: + tstore[key] = blosc2.asarray(data, chunks=chunks, blocks=blocks) server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) - server.subscriber = _Subscriber(urlpath, **kwargs) + server.subscriber = _Subscriber(urlpath, key=key, **kwargs) threading.Thread(target=server.serve_forever, daemon=True).start() urlbase = f"http://127.0.0.1:{server.server_address[1]}/" - array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=kwargs.get("cookie")) + path = f"@public/{name}{key or ''}" + array = blosc2.C2Array(path, urlbase=urlbase, auth_token=kwargs.get("cookie")) return array, server.subscriber, server @@ -503,3 +524,26 @@ def test_a_read_only_cache_is_not_stamped(tmp_path, subscriber): reopened = blosc2.open(cache, mode="r") assert np.array_equal(reopened[0:5, 0:10], data[0:5, 0:10]) + + +def test_blocks_of_a_container_leaf(subscriber, any_chunk_wants_blocks): + """A leaf of a .b2z is a whole frame inside the container, and a subscriber + serves it from that window -- so the client reads its blocks knowing nothing + about containers, which is the whole of what it takes.""" + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), name="tree.b2z", key="/g/leaf") + p = blosc2.Proxy(array, mode="w") + assert array.block_source() is not None + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert not _bytes(sub, "chunk") + assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 8 + assert np.array_equal(p[...], data) + + +def test_a_container_leaf_is_stamped_like_any_other(subscriber): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), name="tree.b2z", key="/g/leaf") + # A leaf has no mtime of its own: the container's is what says it changed + assert array.stamp == f"{sub.mtime}:{sub.array.schunk.cbytes}" From 830f3704cba54a6d57f48a4a8986c4aefddd939c Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:24:10 +0200 Subject: [PATCH 17/25] Record what container-leaf ranges landed as Including the one deviation from the plan: the whole-leaf fetch is served from the window too, because the rebuild it replaced re-partitioned the array and so disagreed with the chunks api/info reports for the same leaf. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 9 +++++ plans/container-leaf-ranges.md | 74 ++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 5bd236902..13fa56cf0 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -64,6 +64,15 @@ XXX version-specific blurb XXX `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. +* `DictStore.member_window(key)` says where a leaf's frame lies inside a `.b2z`, + as `(offset, nbytes)`. A zip store keeps each external leaf uncompressed, so + those bytes are the frame that leaf would have been written as on its own -- + which lets a reader take the window instead of the leaf: Caterva2 now serves a + container leaf from it, so a `Proxy` over `@public/tree.b2z/leaf` reads blocks + exactly as it does over a `.b2nd` (2.8x on a point read, 20x fewer bytes). + None where there is no window: a directory-backed store, an embedded leaf, a + `C2Array` reference. + * `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an earlier run instead of failing on the existing file, so a proxy's cache can outlive the process. The cache must come from a proxy over a source of the same diff --git a/plans/container-leaf-ranges.md b/plans/container-leaf-ranges.md index 8e1ab4d16..29f942f71 100644 --- a/plans/container-leaf-ranges.md +++ b/plans/container-leaf-ranges.md @@ -169,3 +169,77 @@ served` and time the three modes as it does for a plain array. them today (`method=0` throughout), but a `.b2z` produced by another tool could, and its window would decode to nonsense. The member's own frame magic is what catches that, on the server, before the window is offered. + +## What landed + +All four phases, 2026-08-18. blosc2 on `cat2-block-granularity`, Caterva2 on +`range-honesty`. + +| phase | where | commit | +|---|---|---| +| 1. `member_window` | blosc2 `dict_store.py` | *Say where a leaf's frame lies inside a .b2z* | +| 2. leaf ranges | caterva2 `server.py`, `srv_utils.py` | *Serve a container leaf from its window in the file* | +| 3. container fetched whole | caterva2 `server.py` | (same commit) | +| 4. tests | both | (same commits, plus *Pin that a container leaf reads its blocks like any other array*) | + +**The client did not change, at all.** That was the design's claim and it held: +a `Proxy` over `@public/leaves.b2z/big` probes with a `Range` as it does for any +dataset, gets a 206, reads the frame index, and fetches blocks. Nothing in +blosc2 knows that the frame it is reading lives inside a container. + +### One deviation, for a reason that only turned up in the measuring + +The plan said not to serve a *whole* member from its window, on the grounds that +it would change the bytes clients get. It changes them for the better, which the +first measurement showed: the rebuild it replaced went through +`array.slice(..., contiguous=True).to_cframe()`, which **re-partitions**. A leaf +stored with `chunks=(1, 1000, 500)` came back with `chunks=(4, 1000, 500)` while +`api/info` went on reporting the stored ones -- so the two disagreed about the +same leaf, and a client caching that got a partitioning the source never had. +Serving the window makes them agree, because they are the same bytes. + +That also settled what `Accept-Ranges` may say on that response: `bytes`, since +the ranged and whole views are now one representation. Serving two different +byte streams for one URL and advertising ranges over them would have been the +kind of thing `If-Range` exists to catch. + +### The numbers + +A leaf of 8 chunks (3.36 MB each, 20 blocks per chunk) against a local server, +with a network put in front of every request (`--latency-ms 45 +--bandwidth-mbs 10`, cat2.cloud's shape), median of 3: + +| pattern | before (chunks only) | after, blocks | after, multipart | +|---|---|---|---| +| point | 1 req, 3.36 MB, 0.410 s | 2 req, 0.17 MB, 0.145 s | 2 req, 0.17 MB, 0.151 s | +| line, last dim | 1 req, 3.36 MB, 0.408 s | 2 req, 0.17 MB, 0.140 s | 2 req, 0.17 MB, 0.142 s | +| line, first dim | 8 req, 26.92 MB, 0.454 s | 16 req, 1.35 MB, 0.184 s | 2 req, 1.35 MB, 0.271 s | +| window (1/64 per dim) | 1 req, 3.36 MB, 0.413 s | 2 req, 0.17 MB, 0.152 s | 2 req, 0.17 MB, 0.147 s | +| slab (1% of dim 0) | 1 req, 3.36 MB, 0.414 s | 1 req, 3.36 MB, 0.417 s | 1 req, 3.36 MB, 0.415 s | +| slab (10% of dim 0) | 1 req, 3.36 MB, 0.416 s | 1 req, 3.36 MB, 0.414 s | 1 req, 3.36 MB, 0.412 s | + +2.8x on a point or a window, 20x fewer bytes, and the slabs that want every +block of their chunk unchanged -- the same shape a plain `.b2nd` has, which is +the point. Run against the same data stored both ways, the two tables agree row +by row to within the noise. + +The whole-leaf fetch, which now reads the window instead of rebuilding it: + +| | time | bytes | chunks it returns | +|---|---|---|---| +| before | 0.034 s | 26.94 MB | (4, 1000, 500), where `api/info` said (1, 1000, 500) | +| after | 0.016 s | 26.92 MB | (1, 1000, 500), as `api/info` says | + +### Left undone + +- **Only `.b2z` leaves.** An HDF5 dataset is not a Blosc2 frame and a `.b2d` + store keeps its leaves in files of their own (which could be served directly, + and are not today: `api/fetch` on a `.b2d` member is untested ground). +- **An embedded leaf stays whole-chunked.** Its bytes live inside the store's own + compressed super-chunk, so there is no window; `member_window` says so and the + client falls back, which is the arrangement working as intended rather than a + gap to close. +- **`If-Range` / `ETag`.** A leaf rewritten between a client's frame-index read + and its block reads would be read at offsets that no longer mean anything. The + proxy's stamp catches that between *runs* (the container's mtime), not within + one, which is the same exposure a plain `.b2nd` has over any object store. From acf2d24af597a87b4253f70e282450bd80307854 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:24:28 +0200 Subject: [PATCH 18/25] Cross off container-leaf ranges in the block-granularity plan Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 13d62df53..e9d8a3214 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -440,12 +440,12 @@ batched again. ### Left undone -- **A container leaf could serve ranges too.** A TreeStore keeps its leaves as - ordinary frames inside the `.b2z`, so the bytes a block reader wants are in - the file at a fixed offset -- what is missing is a way for the server to say - where a leaf's frame starts, and for the client to add that base to every - range. Worth its own plan; it would give `.b2z` members everything a plain - `.b2nd` has. +- ~~**A container leaf could serve ranges too.**~~ It does now, and it took no + client change at all: see [plans/container-leaf-ranges.md](container-leaf-ranges.md). + The subscriber serves a leaf out of its window in the `.b2z`, so what arrives + is a frame beginning at 0 and everything written for a `.b2nd` applies. 2.8x + on a point read of a leaf, 20x fewer bytes, and the same table as a plain + array row for row. - **Opening a frame could be one request rather than two**, with a suffix range (`Range: bytes=-65536`) batched with the head read. No fsspec backend exposes suffix ranges and `read_ranges` has no way to express one, so it would be a From 438d2fc1b3449e3236a2afec9aba1904dc649edd Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 19:24:51 +0200 Subject: [PATCH 19/25] Shim to reproduce itertools.batched for python 3.11 --- src/blosc2/proxy.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index ce75dd869..06ff731e2 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -17,6 +17,18 @@ from collections.abc import Sequence from concurrent.futures import ThreadPoolExecutor +try: + from itertools import batched +except ImportError: + # Python 3.11 has no itertools.batched + from itertools import islice + + def batched(iterable, n): + it = iter(iterable) + while batch := tuple(islice(it, n)): + yield batch + + try: from numpy.typing import DTypeLike except (ImportError, AttributeError): @@ -720,7 +732,7 @@ def _fetch_by_block(self, item, max_concurrency: int | None): # a batch of range reads (of one, for a transport that takes one) runs = [(n, run) for n in wanted for run in self.src.block_plan(n, wanted[n])] batch = max(getattr(self.src, "max_ranges", 1), 1) - tasks = [((n, None),) for n in whole] + list(itertools.batched(runs, batch)) + tasks = [((n, None),) for n in whole] + list(batched(runs, batch)) # `read_ranges` is the optional half of the protocol: a source that only # has `read_range` is asked one range at a time, as `batch` is 1 for it @@ -1380,7 +1392,7 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: """ section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] - for batch in itertools.batched(todo, max(self.max_ranges, 1)): + for batch in batched(todo, max(self.max_ranges, 1)): spans = [(int(self._offsets[n]), section) for n in batch] heads = self.read_ranges(spans) for nchunk, head in zip(batch, heads, strict=True): From 3c24a11e492d70ac22b2089b37bb7cab4ee2cac0 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 07:43:37 +0200 Subject: [PATCH 20/25] Skip the block tests on wasm, where a socket cannot listen The stand-in subscriber binds a real ThreadingHTTPServer, and Pyodide serves listen(2) out of node's `ws` module -- absent from the cibuildwheel xbuildenv, so the call takes the whole interpreter down as a fatal error rather than raising something a test could catch. Co-Authored-By: Claude Opus 5 --- tests/ndarray/test_c2array_blocks.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 1d9b3c1bf..bd85aca06 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -28,6 +28,11 @@ import blosc2 +# The stand-in subscriber binds a real socket, and Pyodide has no listen(2): +# node asks for the `ws` module that is not there, and takes the runtime down +# with it rather than raising +pytestmark = pytest.mark.skipif(blosc2.IS_WASM, reason="no listening sockets on wasm32") + class _Subscriber: """A Caterva2-shaped server over one .b2nd file.""" From aeb5a1b50641758666f7ed7aa92cb2c6888e3e82 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:23:47 +0200 Subject: [PATCH 21/25] Read a frame's chunk offsets when a chunk is first asked for Opening a frame read the header and the chunk offsets together, which is two requests for something an open does not always need: the header is what says a frame can be read this way at all -- the magic, and the b2nd metalayer whose geometry the source is built from -- while the offsets are only ever wanted once something asks where a chunk is. A Proxy over a cache that already holds the slice wanted asks that of nothing, and paid a round trip for the answer anyway. _read_frame_index splits into _read_frame_header and _read_frame_offsets, and ByteRangeNDSource calls the second on the first chunk touched. _offsets and _extents become properties over a memoized index, so every call site is as it was. Under a lock, since the fetches they serve run in a thread pool: two threads reading the same index is a wasted request rather than a wrong answer, but it is still a wasted request, and one per thread of the first wave. Measured on memory://, requests then bytes, before -> after: opening a 300 KB frame 2/8275 -> 1/8192; blosc2.open(url, lazy=True) the same; a later run over a cache_storage= that holds the slice 2/8275 -> 1/8192. A frame small enough to arrive whole in the first read was already one request and still is. Nothing changes once a chunk is actually fetched -- a corner slice is 4 requests and the same bytes either way -- since there the offsets are read regardless, only later. C2Array is unaffected: block_source() already put the whole open off until a fetch wanted a chunk, so a cache covering the slice cost nothing there before this and costs nothing after. Measured both ways to be sure. The test added for it therefore guards behaviour that already existed, and says so. Three fsspec tests that install their traffic counter after the open now see one read more, because the offsets read moved into the window they count; the wave test sees three waves rather than two, for the same reason. What they were written to prove -- that a wave is batched, not that it is two requests -- is unchanged: four chunks still cost three requests, not three per chunk. Also: the release note claimed a lazy open costs "one request", which is one *read*. FsspecNDSource asks the filesystem for the object's metadata too, which is where its stamp comes from, so its floor is that call plus the header read. C2Array's floor is the single api/info that carries geometry and stamp together. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 10 ++- plans/cat2-block-granularity.md | 10 ++- src/blosc2/proxy.py | 100 ++++++++++++++++++++------- tests/ndarray/test_c2array_blocks.py | 31 ++++++++- tests/test_fsspec.py | 70 ++++++++++++++----- 5 files changed, 176 insertions(+), 45 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 13fa56cf0..2f2d12942 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -60,7 +60,15 @@ XXX version-specific blurb XXX of four (0.237 s → 0.138 s against cat2.cloud), and one for a frame small enough to arrive whole in the first read: the two reads that only measured the next one are guessed at generously instead, since over a network a few hundred - bytes and a few kilobytes cost the same. + bytes and a few kilobytes cost the same. Of those two, only the header is read + when the frame is opened — it is what says the frame can be read this way at + all — and where the chunks are waits for the first chunk anything asks about. + So `blosc2.open(url, lazy=True)` reads once, and so does a whole run over a + `cache_storage=` that already holds the slice wanted: it fetches nothing, and + now asks for no index either. `FsspecNDSource` also asks the filesystem for + the object's metadata, which is where its `stamp` comes from, so its floor is + that call plus the header read; `C2Array` gets geometry and stamp together + from `api/info`, and its floor is that one request. `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index e9d8a3214..08a4ece3b 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -358,7 +358,7 @@ so every existing `Proxy(C2Array(...))` gets blocks without being asked. |---|---|---| | a bench for all of it | blosc2 `bench/ndarray/` | *Measure the Caterva2 block path the way the fsspec one is measured* | | chunks of a container leaf | caterva2 `server.py` | *Serve chunks of a container leaf, and refuse the ones that are not stored* | -| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four* | +| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four*, then *Read a frame's chunk offsets when a chunk is first asked for* | | a stamp for `C2Array` | blosc2 `c2array.py` | *Notice a remote array that was replaced under a proxy's cache* | - **`bench/ndarray/cat2-block-granularity.py`** is the fsspec bench's question @@ -405,6 +405,14 @@ which is the threshold declining to take a chunk apart. Around it: |---|---|---| | one request (`api/info`) | 163.3 ms, a client per request | **42.3 ms**, pooled (3.9x) | | opening a `C2Array` | 0.237 s, 4 requests, 303 bytes | **0.138 s**, 2 requests, 8306 bytes | + +Of those two requests only the first is an open: the header says whether the +frame can be read this way, and the offsets wait for the first chunk anything +asks about. That is worth a request on every `blosc2.open(url, lazy=True)`, and +a whole run's traffic where a `cache_storage=` already holds the slice wanted. +For a `C2Array` it moves a request rather than saving one -- `block_source()` +already put the whole open off until a fetch wanted a chunk, so a cache that +covers the slice cost nothing there before this and costs nothing after. | a wave of 32 ranges | 1.530 s one at a time, 0.208 s eight at a time | **0.136 s**, one multipart request | Phase 4 was built because that last row said to. Starlette *sorts and merges* diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 06ff731e2..706c2032b 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -13,6 +13,7 @@ import os import struct import textwrap +import threading from abc import ABC, abstractmethod from collections.abc import Sequence from concurrent.futures import ThreadPoolExecutor @@ -1127,23 +1128,24 @@ def _chunk_payloads(chunk: bytes, nblocks: int, wanted) -> dict[int, bytes]: return {int(n): chunk[bstarts[n] : bstarts[n] + extents[n]] for n in wanted} -def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: - """Read the header and the chunk offsets of a contiguous frame. +def _read_frame_header(read_range) -> tuple[bytes, list, bytes]: + """Read the header of a contiguous frame. *read_range* is ``(offset, size) -> bytes``, so what this costs is round trips. Returns the raw header bytes, the header decoded as the msgpack - array it is, and the absolute position of every chunk. A negative position - is not a position at all: it encodes a run-length chunk that was never - written to the file. See ``README_CFRAME_FORMAT.rst`` in c-blosc2 for the - layout. - - The format asks to be read in four steps -- how long is the header, the - header, how long is the offsets chunk, the offsets chunk -- and each of the - two questions costs as much as the answer it asks for. So both are guessed - at instead: enough of the head to hold any ordinary header, and the tail - that the frame's own length bounds. A guess that falls short is followed by - the exact read that would have happened anyway, and a frame small enough to - arrive whole in the first read costs one. + array it is, and the prefetched head those came out of -- which + `_read_frame_offsets` needs, since a frame small enough to have arrived + whole carries its offsets in there too. See ``README_CFRAME_FORMAT.rst`` + in c-blosc2 for the layout. + + The format asks how long the header is before handing it over, and the + question costs as much as the answer. So it is guessed at instead: enough + of the head to hold any ordinary header, and a guess that falls short is + followed by the exact read that would have happened anyway. + + This is everything that says whether a frame can be read this way at all -- + the magic, and the metalayer a caller goes on to decode -- so it is what an + open must do eagerly, and the offsets can wait for the first chunk touched. """ import msgpack @@ -1158,11 +1160,24 @@ def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: # bytes, and codec_flags packs clevel into its high nibble: from clevel 8 up # that byte is not valid UTF-8 and decoding the header blows up header = msgpack.unpackb(raw, raw=True, strict_map_key=False) + return raw, header, head + +def _read_frame_offsets(read_range, header: list, head: bytes, header_len: int) -> np.ndarray: + """The absolute position of every chunk of a frame whose header is in hand. + + A negative position is not a position at all: it encodes a run-length chunk + that was never written to the file. + + Like the header before it, the offsets chunk announces its length in an + answer that costs as much as the chunk does, so the tail the frame's own + length bounds is read instead and the announcement checked against it. A + frame that arrived whole in *head* is already past this and costs nothing. + """ # An empty frame has no chunks, so it has no offsets chunk either: what sits # at index_pos is the trailer, and reading it as one fails obscurely if header[8] == 0: # chunksize - return raw, header, np.empty(0, dtype=np.int64) + return np.empty(0, dtype=np.int64) # The offsets live in a Blosc2 chunk of their own, right after the data ones, # so all that follows them is the trailer: the frame's own length says how @@ -1179,7 +1194,7 @@ def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: index = read_range(index_pos, index_cbytes) offsets = np.frombuffer(blosc2.decompress2(index[:index_cbytes]), dtype=np.int64) # Offsets are relative to the end of the header - return raw, header, np.where(offsets >= 0, offsets + header_len, offsets) + return np.where(offsets >= 0, offsets + header_len, offsets) def _frame_metalayer(raw: bytes, header: list, name: str): @@ -1227,9 +1242,13 @@ class ByteRangeNDSource(ProxyNDSource): over HTTP ranges from a Caterva2 subscriber, carrying its auth cookie. A subclass sets its transport up first and then calls this constructor, - which reads the frame's header and chunk offsets through it (three small - reads). It may also set a ``stamp``, anything that names the exact bytes it - reads, so that :ref:`Proxy` can tell a cache built from other bytes. + which reads the frame's header through it -- one small read, and everything + an open decides: that this is a frame at all, and one holding an NDArray. + Where the chunks are waits for the first one anything asks about, so a + :ref:`Proxy` over a cache that already holds the slice wanted opens the + source without ever fetching its index. It may also set a ``stamp``, + anything that names the exact bytes it reads, so that :ref:`Proxy` can tell + a cache built from other bytes. Contiguous frames carrying a ``b2nd`` metalayer only, which is what :func:`blosc2.asarray` and friends write to a single file. Sparse frames and @@ -1264,12 +1283,20 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self.urlpath = urlpath # Exact ranges, not a file handle: a buffered one reads a whole block per # seek (50 MiB on s3fs by default), which would undo the point of a lazy - # open. Chunk reads are stateless, so nothing here is shared between threads - raw, header, self._offsets = _read_frame_index(self.read_range) - self._chunksize = header[8] - self._extents = _chunk_extents(self._offsets, header) + # open. Chunk reads are stateless, so the index below is the only state a + # thread pool shares, and the only thing here that needs a lock + raw, self._header, self._head = _read_frame_header(self.read_range) + self._header_len = len(raw) + self._chunksize = self._header[8] + # Where the chunks are is read on the first one touched, not here: a + # `Proxy` over a cache that already holds the slice asked for fetches + # nothing, and then the offsets are a request spent on nothing at all. + # Everything an open has to decide -- that this is a frame, and one with + # a b2nd metalayer -- is in the header that was just read. + self._index = None + self._index_lock = threading.Lock() try: - _, _, shape, chunks, blocks, dtype_format, dtype = _frame_metalayer(raw, header, "b2nd") + _, _, shape, chunks, blocks, dtype_format, dtype = _frame_metalayer(raw, self._header, "b2nd") except KeyError: raise NotImplementedError( f"{urlpath} has no b2nd metalayer, so it is a plain SChunk rather than an " @@ -1299,6 +1326,31 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): # many short-lived processes ever share one cached array. self._layouts = {} + def _frame_index(self) -> tuple[np.ndarray, np.ndarray]: + """Where every chunk of the frame is, and how much to read at each. + + Read once, on the first chunk anything asks about. Under a lock because + the fetches this serves run in a thread pool: without one the first wave + of them would each read the index, which is a wasted request apiece and + no worse -- what they read is the same either way. + """ + with self._index_lock: + if self._index is None: + offsets = _read_frame_offsets(self.read_range, self._header, self._head, self._header_len) + self._index = (offsets, _chunk_extents(offsets, self._header)) + self._head = None # the prefetch has nothing left to answer + return self._index + + @property + def _offsets(self) -> np.ndarray: + """Where each chunk begins, negative for one that lives in its offset.""" + return self._frame_index()[0] + + @property + def _extents(self) -> np.ndarray: + """How many bytes to read at each chunk's offset to be sure of covering it.""" + return self._frame_index()[1] + @property def shape(self) -> tuple: return self._shape diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index bd85aca06..4467c48db 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -351,6 +351,30 @@ def test_blocks_survive_a_reopened_cache(tmp_path, subscriber, any_chunk_wants_b assert np.array_equal(p[...], data) +def test_a_cache_that_holds_the_slice_costs_no_request(tmp_path, subscriber, any_chunk_wants_blocks): + # Re-running a script over a cache that already covers the slice: `api/info` + # is all it takes. Nothing opens the frame, because opening it is what + # `block_source` puts off until a fetch actually wants a chunk -- and this + # fetch wants none. + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "held.b2nd") + item = (slice(0, 5), slice(0, 10)) + blosc2.Proxy(array, urlpath=cache, mode="a").fetch(item) + + # A later run: its own array over the same subscriber, its own source + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + sub.log.clear() + p = blosc2.Proxy(again, urlpath=cache, mode="a") + p.fetch(item) + assert np.array_equal(p[item], data[item]) + assert not sub.log + + # ... and a slice the cache does not hold opens the frame then + assert np.array_equal(p[100:105, 0:10], data[100:105, 0:10]) + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 4 # head, offsets, layout, block + + def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): # A cache left by a run that fetched whole chunks (which is every run before # this existed) holds complete chunks, so nothing in it is fetched again @@ -395,12 +419,13 @@ def test_a_whole_wave_travels_in_one_request(subscriber, any_chunk_wants_blocks) data = _incompressible((400, 200)) array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) p = blosc2.Proxy(array, mode="w") - assert array.block_source() is not None # the frame index, read once per array + assert array.block_source() is not None # the header, read once per array sub.log.clear() assert np.array_equal(p[:, 0:10], data[:, 0:10]) - # One request for the layouts of all four chunks, one for all their blocks - assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch"] + # One request for where the four chunks are, one for the layouts of all of + # them, one for all their blocks -- three waves, not three per chunk + assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch", "fetch"] assert {status for _, status, _ in sub.log} == {206} assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 4 diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index 3d4f2b625..fdcf19295 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -759,23 +759,29 @@ def _incompressible(shape, chunks, blocks, seed=0): return data, blosc2.asarray(data, chunks=chunks, blocks=blocks) -def test_lazy_open_costs_two_reads(monkeypatch): - # The format asks four questions to find the chunk offsets; both of the two - # that only measure the next read are guessed at instead +def test_lazy_open_costs_one_read(monkeypatch): + # The format asks how long the header is in an answer as dear as the header, + # so the head is guessed at generously instead -- and that one read is the + # whole of an open, since where the chunks are is nothing an open decides data, a = _incompressible((600, 600), (300, 600), (30, 600)) url = _put("openreads.b2nd", a) reads, chunks = _traffic(monkeypatch) src = blosc2.FsspecNDSource(url) - assert len(reads) == 2 + assert len(reads) == 1 assert not chunks assert src.shape == (600, 600) - assert len(src._offsets) == 2 # ... and it read the offsets it came for + + # ... and the offsets are read by the first thing that asks where a chunk is + assert len(src._offsets) == 2 + assert len(reads) == 2 + assert len(src._offsets) == 2 # read once and kept, not once per question + assert len(reads) == 2 -def test_lazy_open_of_a_small_frame_costs_one_read(monkeypatch): +def test_lazy_open_of_a_small_frame_never_reads_twice(monkeypatch): # A frame that fits in the first read is wholly in hand: the offsets chunk - # is in those bytes too, so there is nothing left to ask for + # is in those bytes too, so there is nothing left to ask for, then or later a = blosc2.arange(0, 100, dtype="i4", chunks=(10,)) assert a.schunk.cbytes < blosc2.proxy._FRAME_PREFETCH url = _put("smallframe.b2nd", a) @@ -784,6 +790,7 @@ def test_lazy_open_of_a_small_frame_costs_one_read(monkeypatch): src = blosc2.FsspecNDSource(url) assert len(reads) == 1 assert len(src._offsets) == 10 + assert len(reads) == 1 assert np.array_equal(blosc2.Proxy(src)[:], a[:]) @@ -797,9 +804,34 @@ def test_lazy_open_reads_a_header_that_did_not_fit(monkeypatch): reads, _ = _traffic(monkeypatch) src = blosc2.FsspecNDSource(url) - assert len(reads) == 3 # the guess, the header, the offsets + assert len(reads) == 2 # the guess, then the header assert reads[1] > 4096 assert np.array_equal(blosc2.Proxy(src)[:], data) + assert len(reads) > 2 # the offsets, and the chunks themselves + + +def test_a_cache_that_holds_the_slice_asks_for_no_offsets(monkeypatch, tmp_path): + # What the deferral is for: a later run over a cache that already covers the + # slice fetches nothing, and so has no use for where the chunks are either + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("cachedslice.b2nd", a) + cache = str(tmp_path / "cachedslice-cache.b2nd") + item = (slice(0, 30), slice(0, 600)) + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch(item) + + reads, _ = _traffic(monkeypatch) + src = blosc2.FsspecNDSource(url) + proxy = blosc2.Proxy(src, urlpath=cache, mode="a") + assert len(reads) == 1 # the header, which is what says the frame is readable + + proxy.fetch(item) + assert len(reads) == 1 + assert np.array_equal(proxy[0:30, 0:600], data[0:30, 0:600]) + assert len(reads) == 1 + + # A slice it does not hold pays for the offsets then, and reads right + assert np.array_equal(proxy[300:330, 0:600], data[300:330, 0:600]) + assert len(reads) > 1 def test_lazy_open_reads_an_index_that_did_not_fit(monkeypatch): @@ -811,7 +843,9 @@ def test_lazy_open_reads_an_index_that_did_not_fit(monkeypatch): reads, _ = _traffic(monkeypatch) src = blosc2.FsspecNDSource(url) - assert len(reads) == 3 # the head, the capped tail, the offsets in full + assert len(reads) == 1 # the head, and the offsets not asked for yet + assert len(src._offsets) == 20 + assert len(reads) == 3 # ... then the capped tail, and the offsets in full assert reads[1] == 16 assert np.array_equal(blosc2.Proxy(src)[:], data) @@ -823,10 +857,11 @@ def test_lazy_fetches_only_touched_blocks(monkeypatch): assert cbytes > blosc2.proxy.BLOCK_MIN_CBYTES p = blosc2.open(_put("blocks.b2nd", a), lazy=True) - reads, chunks = _traffic(monkeypatch) # after the open, which reads the frame index + reads, chunks = _traffic(monkeypatch) # after the open, which reads the header assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) - # One read for the block offsets, one for the single block the slice lands in - assert len(reads) == 2 + # One read for where the chunks are, one for the block offsets inside the one + # wanted, one for the single block the slice lands in + assert len(reads) == 3 assert not chunks assert sum(reads) < cbytes / 5 @@ -851,7 +886,9 @@ def test_lazy_whole_array_skips_the_block_path(monkeypatch, any_chunk_wants_bloc reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[:], data) assert len(chunks) == 2 - assert len(reads) == len(chunks) # one request each, none of them for block offsets + # One request per chunk and none for block offsets, after the one that says + # where the chunks are + assert len(reads) == len(chunks) + 1 @pytest.mark.parametrize( @@ -902,7 +939,7 @@ def test_lazy_block_cache_survives_reopen(tmp_path, monkeypatch, any_chunk_wants # A partly filled chunk survives, so the blocks in it do not travel again p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") - reopened = len(reads) # every open reads the frame index afresh + reopened = len(reads) # every open reads the header afresh assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) assert len(reads) == reopened # ... and the ones missing from it still do @@ -931,8 +968,9 @@ def test_lazy_blocks_fall_back_for_memcpyed_chunks(monkeypatch, any_chunk_wants_ reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) assert len(chunks) == 1 - # The offsets are read, say there is nothing to skip, and the chunk follows - assert len(reads) == 2 + # Where the chunks are, then the block offsets, which say there is nothing to + # skip, and the chunk follows + assert len(reads) == 3 def test_lazy_blocks_with_run_length_chunks(monkeypatch, any_chunk_wants_blocks): From 993718883d013c5e54aefae27964fa236e4baa2b Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:38:57 +0200 Subject: [PATCH 22/25] Keep where the chunks and blocks are in the cache A run learned two things about where bytes lie in a remote frame and threw both away when it ended: the chunk offsets, and the block offsets of the chunks it took apart. The next run over the same cache read them again -- the offsets to fetch anything at all, a layout for every chunk it wanted blocks of that a previous run had only half filled. Both now go in the cache's vlmeta, as the bytes they were read as, so what comes back goes through `_parse_layout` exactly as a fresh read does rather than through a second parser that could drift from it. Layouts are kept for the partly filled chunks alone: a fetch asks for a layout only where blocks are missing, so a complete chunk is never asked about again and an untouched one never was, which bounds the blob to what a later fetch could actually use. Only for a source that can name the bytes it read, and only against a cache whose recorded stamp matches. These are positions *in a file*: a frame replaced underneath keeps its geometry while every chunk and every block inside it moves, and blocks spliced at the previous frame's positions decode to nonsense rather than to stale data. The stamp is checked here rather than taken on trust from how the cache was come by, since a `_cache=` handed in never passed `_reopen_cache`. An index that does not fit the source as it stands -- wrong blocks per chunk, wrong number of offsets -- is dropped rather than used. Measured on a 4-chunk frame of 1.36 MB chunks, 100 blocks each, a later run fetching different blocks of chunks a previous run half filled: 4 requests to 2 against the subscriber stand-in (the header and the blocks, nothing between), 46 to 40 against fsspec, where a layout is a request per chunk rather than one for all of them. On a 200-chunk array `_save_fetched` takes 0.53 ms with the index in it, against a round trip of tens of milliseconds saved. `C2Array` takes the index but does not hand it on until there is a source to hand it to: passing it through `block_source()` would build the source to receive it, and building one reads the frame's header -- a request, at the very moment of a run that may go on to fetch nothing at all. A cache with no `proxy-index` entry reads as it did, so nothing already on disk needs rebuilding; `proxy-index` joins the vlmeta keys a caller may not set. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 10 ++- plans/cat2-block-granularity.md | 11 +++- src/blosc2/c2array.py | 25 ++++++- src/blosc2/proxy.py | 99 ++++++++++++++++++++++++++-- tests/ndarray/test_c2array_blocks.py | 42 +++++++++++- tests/test_fsspec.py | 70 ++++++++++++++++++++ 6 files changed, 245 insertions(+), 12 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 2f2d12942..573cdb9b9 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -68,7 +68,15 @@ XXX version-specific blurb XXX now asks for no index either. `FsspecNDSource` also asks the filesystem for the object's metadata, which is where its `stamp` comes from, so its floor is that call plus the header read; `C2Array` gets geometry and stamp together - from `api/info`, and its floor is that one request. + from `api/info`, and its floor is that one request. A persisted cache keeps + what the source read about *where* things are — the frame's chunk offsets, and + the block offsets of the chunks it holds only part of — so a later run over it + starts from those instead of reading them again. A warm fetch of blocks missing + from a chunk already half held goes from 4 requests to 2 against a subscriber, + and drops the offsets read and one layout read per chunk touched against an + object store. Only for a source that can name the bytes it read: positions in a + frame are worth nothing against a frame that was replaced, so an unstamped + source keeps none of this and reads as before. `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 08a4ece3b..2c6db7d65 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -358,7 +358,7 @@ so every existing `Proxy(C2Array(...))` gets blocks without being asked. |---|---|---| | a bench for all of it | blosc2 `bench/ndarray/` | *Measure the Caterva2 block path the way the fsspec one is measured* | | chunks of a container leaf | caterva2 `server.py` | *Serve chunks of a container leaf, and refuse the ones that are not stored* | -| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four*, then *Read a frame's chunk offsets when a chunk is first asked for* | +| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four*, then *Read a frame's chunk offsets when a chunk is first asked for* and *Keep where the chunks and blocks are in the cache* | | a stamp for `C2Array` | blosc2 `c2array.py` | *Notice a remote array that was replaced under a proxy's cache* | - **`bench/ndarray/cat2-block-granularity.py`** is the fsspec bench's question @@ -413,6 +413,15 @@ a whole run's traffic where a `cache_storage=` already holds the slice wanted. For a `C2Array` it moves a request rather than saving one -- `block_source()` already put the whole open off until a fetch wanted a chunk, so a cache that covers the slice cost nothing there before this and costs nothing after. + +The cache then keeps both indexes it read -- the frame's chunk offsets, and the +block offsets of the chunks it holds only part of -- under the same stamp that +guards the cached chunks themselves. A warm fetch of blocks missing from a +half-held chunk: 4 requests to 2 against the subscriber stand-in (header and +blocks, nothing between), 46 to 40 against fsspec, where the layout wave is a +request per chunk rather than one for all of them. Layouts are kept for the +partly filled chunks alone, which bounds the blob to what a later fetch could +ask about: a complete chunk is never asked again, an untouched one never was. | a wave of 32 ranges | 1.530 s one at a time, 0.208 s eight at a time | **0.136 s**, one multipart request | Phase 4 was built because that last row said to. Starlette *sorts and merges* diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index 71a76777d..972607db5 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -452,6 +452,8 @@ def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | N # dataset cannot be read in ranges) or a C2NDSource self._block_source = _UNTRIED self._block_lock = threading.Lock() + # An index a `Proxy` handed over before the source existed; see adopt_index + self._pending_index = None # Try to 'open' the remote path try: @@ -718,12 +720,33 @@ def _open_block_source(self) -> C2NDSource | None: # something only the answer to a range request can say: an HDF5 leaf or a # `.b2z` member reports one and is streamed all the same try: - return C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) + source = C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) + source.adopt_index(self._pending_index) + return source except (_NotRanged, ValueError, NotImplementedError, RuntimeError, _httpx().HTTPError): # Not ranged, not a contiguous frame, not an NDArray, or not # reachable: whole chunks work for all of those return None + def adopt_index(self, state) -> None: + """Keep an index a `Proxy` read out of its cache until there is a source. + + Handing it straight to :meth:`block_source` would build the source to + receive it, and building it reads the frame's header -- a request, at the + very moment of a run that may go on to fetch nothing at all. So it waits + here, and `_open_block_source` passes it on to the source it builds. + """ + self._pending_index = state + + def index_state(self, keep=()) -> dict | None: + """What a `Proxy` should keep of what was read; see :ref:`ByteRangeNDSource`.""" + source = self._block_source + if source is _UNTRIED or source is None: + # No source was ever built, so nothing was read through one: hand back + # whatever came out of the cache, rather than dropping it + return self._pending_index + return source.index_state(keep) + def wants_blocks(self, nchunk: int, nwanted: int) -> bool: """Whether fetching *nwanted* blocks of a chunk beats fetching all of it.""" source = self.block_source() diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 706c2032b..2da6ad360 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -85,7 +85,9 @@ def batched(iterable, n): # vlmeta entries the proxy keeps its own state in: what it has fetched, and which # remote bytes the cache was filled from. A caller cannot write these. -_RESERVED_VLMETA = frozenset({"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "proxy-stamp"}) +_RESERVED_VLMETA = frozenset( + {"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "proxy-stamp", "proxy-index"} +) # `jit` kwargs that tune *how* an expression is evaluated, not what container the # result is stored in. Unlike storage kwargs (`cparams`, `chunks`, `urlpath`, ...), @@ -415,6 +417,18 @@ def __init__( # hands over for a persisted proxy: nothing may be written there, and a # proxy over one stays observational anyway (see `__getitem__`) self._schunk_cache.vlmeta["proxy-stamp"] = stamp + # Where the chunks are, and where the blocks of the partly filled ones + # are, as an earlier run read them. Only from a cache that names the very + # same remote bytes, checked here rather than taken on trust from how the + # cache was come by: a `_cache=` handed in never passed `_reopen_cache`. + adopt = getattr(self.src, "adopt_index", None) + if ( + adopt is not None + and not fresh + and stamp is not None + and self._schunk_cache.vlmeta.get("proxy-stamp") == stamp + ): + adopt(self._schunk_cache.vlmeta.get("proxy-index")) if vlmeta: reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) if reserved: @@ -563,6 +577,27 @@ def _save_fetched(self) -> None: self._schunk_cache.vlmeta[self._fetched_key] = bytes(self._fetched) if self._blocks_per_chunk > 1: self._schunk_cache.vlmeta["proxy-fetched-bpc"] = self._blocks_per_chunk + # Where the source read things to be, so the next run over this cache need + # not ask again. Only for a source that can name the bytes it read: an + # unstamped one cannot tell a replaced frame from the one these positions + # came from, and reusing them across a replacement is worse than serving + # stale data. Bounded by keeping layouts for the partly filled chunks + # alone, which are the only ones a later fetch would ask about. + state = getattr(self.src, "index_state", None) + if state is not None and getattr(self.src, "stamp", None) is not None: + self._schunk_cache.vlmeta["proxy-index"] = state(self._partly_filled()) + + def _partly_filled(self) -> list[int]: + """Chunks the cache holds some of the blocks of, but not all.""" + bpc = self._blocks_per_chunk + nchunks = self._schunk_cache.nchunks + if bpc == 1 or not nchunks: + return [] + # Counted over the whole bitmap at once: a loop over chunks x blocks is + # millions of bit tests on a large array, and this runs after every fetch + bits = np.unpackbits(np.frombuffer(bytes(self._fetched), dtype=np.uint8), bitorder="little") + counts = bits[: nchunks * bpc].reshape(nchunks, bpc).sum(axis=1) + return np.flatnonzero((counts > 0) & (counts < bpc)).tolist() def _reopen_cache(self, urlpath: str): """Adopt the cache container stored at *urlpath*, checking it fits the source.""" @@ -1318,13 +1353,62 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): if all(self._blocks) else 1 ) - # ponytail: layouts are memoized for the life of the source, so a second - # slice pays no header read. Persisting them in the cache would save one - # round trip more, but only for a new process reaching into a chunk it - # had partly explored before -- every other repeat already skips the read, - # since a fetch asks for layouts only where blocks are missing. Do it if - # many short-lived processes ever share one cached array. + # Layouts are memoized for the life of the source, and `_sections` keeps + # the bytes each was read as, so a `Proxy` can hand them to its cache and + # a later run start from them instead of reading them again. self._layouts = {} + self._sections = {} + + def index_state(self, keep: Sequence[int] = ()) -> dict: + """Where things are, as the bytes they were read as, for a cache to keep. + + The frame's chunk offsets, and the header sections of the chunks in + *keep*. A layout is worth keeping only for a chunk the cache holds some + but not all of: a fetch asks for layouts only where blocks are missing, + so a chunk that is complete is never asked about again, and one that is + empty was never read. + + Kept as read rather than as parsed, so that what comes back goes through + :meth:`_parse_layout` exactly as a fresh read does -- one parser, not a + second one that could disagree with it. + """ + offsets = self._index[0] if self._index is not None else None + return { + "bpc": self.blocks_per_chunk, + "offsets": b"" if offsets is None else offsets.tobytes(), + "layouts": [[n, self._sections[n]] for n in keep if n in self._sections], + } + + def adopt_index(self, state: dict | None) -> None: + """Take up what an earlier run left behind in :meth:`index_state`. + + Only ever called with a state saved against the very same remote bytes -- + :ref:`Proxy` checks the source's ``stamp`` against the one its cache + recorded first -- and that is what makes these safe to reuse. They are + positions *in a file*: a frame replaced underneath keeps its geometry + while every chunk and every block inside it moves, and blocks spliced at + the positions of the frame before it decode to nonsense rather than to + stale data. Anything that does not fit the source as it stands now is + dropped rather than trusted. + """ + if not state or self._index is not None: + return + if state.get("bpc") != self.blocks_per_chunk: + return + section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk + offsets = state.get("offsets") or b"" + if offsets: + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self._shape, self._chunks, strict=True)) + array = np.frombuffer(offsets, dtype=np.int64) + if len(array) != nchunks: + return + with self._index_lock: + self._index = (array, _chunk_extents(array, self._header)) + self._head = None # the prefetch has nothing left to answer + for nchunk, head in state.get("layouts") or (): + if len(head) <= section: # what a read of one asks for, or a short answer + self._sections[nchunk] = head + self._layouts[nchunk] = self._parse_layout(head, section) def _frame_index(self) -> tuple[np.ndarray, np.ndarray]: """Where every chunk of the frame is, and how much to read at each. @@ -1448,6 +1532,7 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: spans = [(int(self._offsets[n]), section) for n in batch] heads = self.read_ranges(spans) for nchunk, head in zip(batch, heads, strict=True): + self._sections[nchunk] = head self._layouts[nchunk] = self._parse_layout(head, section) return [self._layouts[n] for n in nchunks] diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 4467c48db..31a636bf1 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -370,9 +370,47 @@ def test_a_cache_that_holds_the_slice_costs_no_request(tmp_path, subscriber, any assert np.array_equal(p[item], data[item]) assert not sub.log - # ... and a slice the cache does not hold opens the frame then + # ... and a slice the cache does not hold opens the frame then: the header, + # the layout of the chunk it lands in, and the blocks. Not where the chunks + # are -- the earlier run left that in the cache assert np.array_equal(p[100:105, 0:10], data[100:105, 0:10]) - assert [kind for kind, _, _ in sub.log] == ["fetch"] * 4 # head, offsets, layout, block + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 3 + + +def test_a_kept_index_halves_a_warm_fetch(tmp_path, subscriber, any_chunk_wants_blocks): + # A later run wanting different blocks of chunks a previous one half filled: + # where the chunks are and where those blocks are both came out of the cache, + # so what travels is the header and the blocks, and nothing between + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "kept.b2nd") + blosc2.Proxy(array, urlpath=cache, mode="a").fetch((slice(None), slice(0, 10))) + + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + sub.log.clear() + p = blosc2.Proxy(again, urlpath=cache, mode="a") + assert np.array_equal(p[:, 100:110], data[:, 100:110]) + assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch"] # the header, the blocks + assert np.array_equal(p[...], data) # ... and the rest still reads right + + +def test_a_kept_index_does_not_open_the_frame_to_be_taken_up(tmp_path, subscriber, any_chunk_wants_blocks): + # Handing the index to the source would build the source to receive it, and + # building it reads the header -- a request, at the very moment of a run that + # may go on to fetch nothing. It waits with the array until there is a source + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "unopened.b2nd") + item = (slice(0, 5), slice(0, 10)) + blosc2.Proxy(array, urlpath=cache, mode="a").fetch(item) + + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + sub.log.clear() + p = blosc2.Proxy(again, urlpath=cache, mode="a") + assert again._pending_index is not None # taken out of the cache, not yet used + assert not sub.log + p.fetch(item) + assert not sub.log def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index fdcf19295..7e36edaa2 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -810,6 +810,76 @@ def test_lazy_open_reads_a_header_that_did_not_fit(monkeypatch): assert len(reads) > 2 # the offsets, and the chunks themselves +def test_a_cache_keeps_where_the_chunks_and_blocks_are(monkeypatch, tmp_path): + # A later run over a partly filled cache asks for blocks it has not got, and + # for nothing else: where the chunks are, and where the blocks inside the + # chunks it half holds are, both came out of the cache + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("keptindex.b2nd", a) + cache = str(tmp_path / "keptindex-cache.b2nd") + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch((slice(0, 30), slice(None))) + + holder = blosc2.open(cache) + state = holder.schunk.vlmeta["proxy-index"] + assert len(state["offsets"]) == 2 * 8 # one int64 per chunk of the frame + assert [n for n, _ in state["layouts"]] == [0] # the one chunk half held + del holder + + reads, _ = _traffic(monkeypatch) + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + assert len(reads) == 1 # the header, which an open reads whatever else it knows + + # Different blocks of the chunk already half held: only they travel + assert np.array_equal(p[60:90, 0:10], data[60:90, 0:10]) + assert len(reads) == 2 + assert np.array_equal(p[...], data) # ... and the rest still reads right + + +def test_a_cache_keeps_nothing_for_a_source_that_cannot_name_its_bytes(tmp_path, monkeypatch): + # Positions in a file are only reusable against the same file, and a source + # without a stamp cannot say it is the same one. Nothing is kept for it, + # rather than kept and hoped for + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("unstamped.b2nd", a) + cache = str(tmp_path / "unstamped-cache.b2nd") + src = blosc2.FsspecNDSource(url) + src.stamp = None # as a source that cannot name its bytes leaves it + blosc2.Proxy(src, urlpath=cache, mode="w").fetch((slice(0, 30), slice(None))) + + holder = blosc2.open(cache) + assert "proxy-index" not in holder.schunk.vlmeta + del holder + + reads, _ = _traffic(monkeypatch) + src = blosc2.FsspecNDSource(url) + src.stamp = None + p = blosc2.Proxy(src, urlpath=cache, mode="a") + assert np.array_equal(p[60:90, 0:10], data[60:90, 0:10]) + assert len(reads) > 2 # the header, and the offsets and layout it kept nothing of + assert np.array_equal(p[...], data) + + +def test_a_kept_index_of_the_wrong_shape_is_dropped(tmp_path, monkeypatch): + # Belt to the stamp's braces: whatever else went wrong, an index that does not + # fit the source as it stands is read afresh rather than used + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("wrongindex.b2nd", a) + cache = str(tmp_path / "wrongindex-cache.b2nd") + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch((slice(0, 30), slice(None))) + + holder = blosc2.open(cache, mode="a") + state = dict(holder.schunk.vlmeta["proxy-index"]) + state["offsets"] = state["offsets"][:8] # one chunk's worth, for a frame of two + holder.schunk.vlmeta["proxy-index"] = state + del holder + + reads, _ = _traffic(monkeypatch) + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + assert np.array_equal(p[60:90, 0:10], data[60:90, 0:10]) + assert len(reads) > 2 # the offsets were read again + assert np.array_equal(p[...], data) + + def test_a_cache_that_holds_the_slice_asks_for_no_offsets(monkeypatch, tmp_path): # What the deferral is for: a later run over a cache that already covers the # slice fetches nothing, and so has no use for where the chunks are either From 95850a8138d4fab3fc9ff048447675d0d2c16a75 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:51:46 +0200 Subject: [PATCH 23/25] Count the reads an S3 block fetch makes now, and pin the kept index `test_lazy_block_reads` installs its hook after the open, so the offsets read -- which used to happen at the open and now waits for the first chunk asked about -- landed inside the window it counts. Three reads rather than two: where the chunks are, where the blocks of the one wanted are, and the block. The bytes it bounds are unchanged, since the same reads happen either way. Only this file was left, because s3fs and moto are not in the default test environment; the same shift in the memory:// tests went with the commit that caused it. While here, two tests for the index the cache now keeps, in the file that exists for what memory:// cannot show. `ukey` on a real store is an object identity rather than a size, so this is where a replaced object can be checked honestly: one test replaces a frame with another of the same shape and partitioning and holds that the cache is refused, since every offset in it now points elsewhere in a frame of the same size -- a stale chunk serves old data, a stale offset serves nonsense. The other holds that a later run over a half-filled chunk reads one block and nothing else: no offsets, no layout, both out of the cache. Co-Authored-By: Claude Opus 5 --- tests/test_fsspec_s3.py | 59 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/tests/test_fsspec_s3.py b/tests/test_fsspec_s3.py index b6cf98522..001cef1d9 100644 --- a/tests/test_fsspec_s3.py +++ b/tests/test_fsspec_s3.py @@ -137,8 +137,63 @@ def test_lazy_block_reads(blocky, max_concurrency): p.src.read_range = lambda *args: (out := original(*args), traffic.append(len(out)))[0] assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) - # The offsets and one block, against 1.4 MB of chunk - assert len(traffic) == 2 + # Where the chunks are, where the blocks of the one wanted are, and one + # block, against 1.4 MB of chunk. The frame's header is not in here: the + # open read that, before the hook went on + assert len(traffic) == 3 assert sum(traffic) < 200_000 # And the array still reads back whole, over the blocks already cached assert np.array_equal(p[...], data) + + +def test_a_kept_index_is_refused_when_the_object_was_replaced(s3_endpoint, tmp_path): + """The cache keeps *positions* now, which a replaced object invalidates. + + A stale chunk cache serves old data; a stale index is worse, since blocks + fetched at the old frame's offsets and spliced into a chunk decode to + nonsense. s3fs is where this can be checked honestly: its `ukey` is a real + object identity, where `memory://` has almost no metadata to build one from. + """ + data = np.random.default_rng(0).random((600, 600)) + url = f"s3://{BUCKET}/replaced.b2nd" + blosc2.asarray(data, chunks=(300, 600), blocks=(30, 600)).save(url, mode="w") + cache = str(tmp_path / "replaced-cache.b2nd") + + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) + del p + holder = blosc2.open(cache) + assert "proxy-index" in holder.schunk.vlmeta # where the chunks and blocks are + del holder + + # Same shape, same partitioning, other bytes: every offset in there now + # points somewhere else in a frame of the same size + other = np.random.default_rng(1).random((600, 600)) + blosc2.asarray(other, chunks=(300, 600), blocks=(30, 600)).save(url, mode="w") + with pytest.raises(ValueError, match="different remote bytes"): + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + + # ... and starting afresh reads the new bytes, index and all + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w") + assert np.array_equal(p[10:12, 30:40], other[10:12, 30:40]) + assert np.array_equal(p[...], other) + + +def test_a_kept_index_spares_a_later_run_the_reads(s3_endpoint, tmp_path): + # The same slice geometry as test_lazy_block_reads, one run later: the + # offsets and the layout of the half-held chunk both come out of the cache, + # so only the blocks that are missing travel + data = np.random.default_rng(0).random((600, 600)) + url = f"s3://{BUCKET}/kept.b2nd" + blosc2.asarray(data, chunks=(300, 600), blocks=(30, 600)).save(url, mode="w") + cache = str(tmp_path / "kept-cache.b2nd") + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a").fetch((slice(10, 12), slice(30, 40))) + + src = blosc2.FsspecNDSource(url) + traffic = [] + original = src.read_range + src.read_range = lambda *args: (out := original(*args), traffic.append(len(out)))[0] + p = blosc2.Proxy(src, urlpath=cache, mode="a") + assert np.array_equal(p[60:62, 30:40], data[60:62, 30:40]) + assert len(traffic) == 1 # one block, and nothing to say where it was + assert np.array_equal(p[...], data) From eb7fd15050dfd6cd5817693a464b8cd61a250f65 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:57:40 +0200 Subject: [PATCH 24/25] Stop the subscriber stand-in idling half a second per test Each test in this file starts a server and shuts it down, and `shutdown()` waits for `serve_forever`'s poll interval to elapse before the loop notices it was asked to stop. At the default half second, and with 27 tests, that was 13.5 s of a 14.5 s file spent waiting for nothing -- enough to make it the slowest file in the suite, and under `--dist loadfile` the slowest file is the wall clock. A hundredth of a second instead: the file goes from 14.45 s to 0.98 s and out of the top ten, and the whole suite from 26 s to 21 s. Nothing about what is tested changes; the poll interval only decides how often an idle serve loop looks up. Co-Authored-By: Claude Opus 5 --- tests/ndarray/test_c2array_blocks.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 31a636bf1..55f09bf1c 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -183,7 +183,11 @@ def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", key=None, **kwargs): tstore[key] = blosc2.asarray(data, chunks=chunks, blocks=blocks) server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) server.subscriber = _Subscriber(urlpath, key=key, **kwargs) - threading.Thread(target=server.serve_forever, daemon=True).start() + # A short poll interval, because `shutdown()` waits for one to elapse before + # the serve loop notices: at the default 0.5 s that is half a second of doing + # nothing per test, and this file has enough of them for that to be most of + # what it costs + threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True).start() urlbase = f"http://127.0.0.1:{server.server_address[1]}/" path = f"@public/{name}{key or ''}" array = blosc2.C2Array(path, urlbase=urlbase, auth_token=kwargs.get("cookie")) From 31a84fc7cc37eb06098e23d9e6d7d49c93b067d1 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 00:32:09 +0200 Subject: [PATCH 25/25] Check the cache's stamp, and the bytes a range answer actually carries A review of the block-granularity branch turned up a handful of ways the block path takes an answer on trust: - The stale-index guard compared the stamp with the copy of itself the constructor had just written, so it passed for every writable cache. A `_cache=` handed in never passes `_reopen_cache`, which is where the other check lives, so a replaced remote frame had the old chunk offsets adopted and its blocks spliced at positions that mean nothing now. - `_span_of` checked only that a part *began* before the span it was asked for, so a part ending early sliced short and the payload went into a chunk whose `bstarts` promises the full length. - A 206 without a Content-Range, or a multipart answer without a boundary, raised KeyError/IndexError out of `Proxy.fetch` instead of being refused as an answer that cannot be placed. - A 503 to the one-time probe was written down as "this dataset is streamed" for the life of the array, though nothing was downloaded to find that out. Transient failures now leave the source untried, while a streamed 200 is still an answer for good. - The frame behind a C2Array was never checked against the geometry `api/info` reported for it, though a Proxy takes the block grid from one and the `bstarts` from the other. - `member_window` handed out the *uncompressed* size of a zip member and never checked the member was stored, so a repacked `.b2z` would have had deflated bytes served as a frame. The shared HTTP client's forgetful cookie jar was inert, since httpx re-wraps an `httpx.Cookies` subclass in a plain one and drops the override -- a `Set-Cookie` from any subscriber was kept and replayed to every other. A bare `CookieJar` is handed through instead. And the kept index gets the same treatment: the offsets are written little-endian rather than however this host stores them, a stored header section is adopted only at exactly the length a read of one asks for (a short one parsed to None and pinned that chunk to whole-chunk fetches for good, or tripped over `struct.unpack` if it was really short), and it is written back only when it says something new instead of after every fetch that moved a byte. Two reads that were paying for it: the frame index is taken once per call rather than once per element under its lock, and layout batches past the first are overlapped like any other request rather than run one after the next. Co-Authored-By: Claude Opus 5 --- src/blosc2/c2array.py | 133 ++++++++++++++++++++++----- src/blosc2/dict_store.py | 15 ++- src/blosc2/proxy.py | 63 +++++++++---- tests/ndarray/test_c2array_blocks.py | 68 +++++++++++++- tests/test_fsspec.py | 28 ++++++ 5 files changed, 260 insertions(+), 47 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index 972607db5..e9260819f 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -10,6 +10,7 @@ import atexit import math import os +import struct import threading from contextlib import contextmanager from typing import TYPE_CHECKING @@ -49,7 +50,7 @@ def _httpx(): _client_lock = threading.Lock() -def _forgetful_cookies(httpx): +def _forgetful_cookies(): """A cookie jar that never keeps anything, for the shared client. A client of its own per request could not carry a cookie from one request to @@ -57,10 +58,15 @@ def _forgetful_cookies(httpx): being read, and arrays with different tokens (or none) share this client. A `Set-Cookie` from any response would otherwise start authorizing requests that asked for none. + + A jar rather than an `httpx.Cookies` subclass: the client re-wraps whatever + it is handed in a plain `httpx.Cookies`, which copies the cookies over and + drops the subclass, but hands a bare `CookieJar` straight through. """ + import http.cookiejar - class _NoCookies(httpx.Cookies): - def extract_cookies(self, response): + class _NoCookies(http.cookiejar.CookieJar): + def extract_cookies(self, response, request): pass return _NoCookies() @@ -90,7 +96,7 @@ def _sync_client(): _client = httpx.Client( timeout=TIMEOUT, limits=httpx.Limits(max_connections=64, max_keepalive_connections=32), - cookies=_forgetful_cookies(httpx), + cookies=_forgetful_cookies(), ) return _client @@ -276,14 +282,36 @@ def slice_to_string(slice_): class _NotRanged(Exception): """The subscriber answered a range request with something other than a 206.""" + def __init__(self, message: str, status: int | None = None): + super().__init__(message) + self.status = status + + @property + def transient(self) -> bool: + """Whether asking again could be answered differently. + + A 200 is the dataset itself, streamed, and no amount of asking again will + make it a file; a server that is busy or broken says nothing at all about + how the dataset is served, and costs no download to ask twice. + """ + return self.status is not None and (self.status >= 500 or self.status == 429) + class _PartsMissing(Exception): """A multi-range answer did not carry all the bytes that were asked for.""" def _content_range(value: str) -> int: - """Where a `Content-Range: bytes start-end/total` header says its part starts.""" - return int(value.split()[1].split("-")[0]) + """Where a `Content-Range: bytes start-end/total` header says its part starts. + + Malformed answers are refused rather than guessed at: `bytes */1234` for a + range that could not be satisfied has no start to read, and a part written to + no shape at all cannot be placed in the frame. + """ + try: + return int(value.split()[1].split("-")[0]) + except (IndexError, ValueError) as exc: + raise _NotRanged(f"a 206 carried an unreadable Content-Range: {value!r}") from exc def _byteranges(response) -> list[tuple[int, bytes]]: @@ -296,8 +324,17 @@ def _byteranges(response) -> list[tuple[int, bytes]]: """ content_type = response.headers.get("content-type", "") if "multipart/byteranges" not in content_type: - return [(_content_range(response.headers["content-range"]), response.content)] - boundary = content_type.split("boundary=")[1].strip().strip('"').encode() + # Where the part sits is the one thing the body cannot say, so an answer + # without it is refused: a caching proxy that strips the header would + # otherwise have its bytes placed wherever they were asked for + single = response.headers.get("content-range") + if single is None: + raise _NotRanged("a 206 arrived without a Content-Range to place it by") + return [(_content_range(single), response.content)] + _, sep, boundary = content_type.partition("boundary=") + if not sep: + raise _NotRanged(f"a multipart answer named no boundary: {content_type!r}") + boundary = boundary.strip().strip('"').encode() parts = [] for chunk in response.content.split(b"--" + boundary): head, sep, body = chunk.partition(b"\r\n\r\n") @@ -315,7 +352,10 @@ def _byteranges(response) -> list[tuple[int, bytes]]: def _span_of(parts: list[tuple[int, bytes]], offset: int, size: int, url: str) -> bytes: """The bytes of one requested span, out of whichever part covers it.""" for start, data in parts: - if start <= offset < start + len(data): + # The whole span, not just its first byte: a part that begins inside it + # and ends early would otherwise slice short, and a short block payload + # is spliced against a `bstarts` that promises the full length + if start <= offset and offset + size <= start + len(data): return data[offset - start : offset - start + size] raise _PartsMissing(f"{url} answered without the bytes at {offset}, which were asked for") @@ -344,6 +384,19 @@ def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY self._url = _sub_url(array.urlbase, f"api/fetch/{array.path}") self._auth_token = array.auth_token super().__init__(self._url, max_concurrency) + # A `Proxy` mixes the two: the block grid and the fetched bitmap come from + # the array's `api/info`, while the header sections and `bstarts` come from + # this frame. They have to be the same dataset for that to mean anything, + # and the magic bytes alone do not say so -- a window off by a member of a + # `.b2z`, or a path serving a file other than the one described, reads a + # frame that parses and splices blocks into chunks of the wrong shape. + # Geometry alone, since that is what the block arithmetic on both sides is + # built out of, and a dtype `api/info` reports as a repr would fail to + # parse here for a dataset that reads perfectly well + described = (tuple(array.shape), tuple(array.chunks), tuple(array.blocks)) + found = (tuple(self._shape), tuple(self._chunks), tuple(self._blocks)) + if described != found: + raise ValueError(f"{self._url} serves {found}, where its dataset is {described}") def read_range(self, offset: int, size: int) -> bytes: return self._get([(offset, size)])[0] @@ -382,7 +435,10 @@ def _get(self, spans: list[tuple[int, int]]) -> list[bytes]: # Whatever this is, it is not the bytes that were asked for: a 200 # carries the whole dataset, which is the download this exists to # avoid, so leave the body unread on the socket - raise _NotRanged(f"{self._url} answered {response.status_code} to a Range request") + raise _NotRanged( + f"{self._url} answered {response.status_code} to a Range request", + response.status_code, + ) response.read() parts = _byteranges(response) return [_span_of(parts, offset, size, self._url) for offset, size in spans] @@ -701,31 +757,58 @@ def block_source(self) -> C2NDSource | None: with self._block_lock: if self._block_source is _UNTRIED: self._block_source = self._open_block_source() - return self._block_source + # A failure that says nothing about the dataset leaves it _UNTRIED, so the + # next fetch asks again; this one keeps to whole chunks either way + return None if self._block_source is _UNTRIED else self._block_source - def _open_block_source(self) -> C2NDSource | None: - """Decide, at whatever cost it takes, whether this dataset serves ranges.""" + def _open_block_source(self): + """Decide, at whatever cost it takes, whether this dataset serves ranges. + + None for a dataset that does not serve ranges, which is an answer for + good; `_UNTRIED` for a subscriber that could not say, which is not. + """ + httpx = _httpx() # `api/info` rules out a dataset the subscriber computes for nothing: a # stored one reports its geometry where a lazy expression reports # `expression` and `operands` if not all(key in self.meta for key in ("chunks", "blocks", "schunk")): return None - # Nor is a frame of small chunks worth an index read: blosc2 declines to - # take a chunk below BLOCK_MIN_CBYTES apart, so nothing here would ever - # use a block, and the dataset keeps exactly the behaviour it had before - nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) - if not nchunks or self.cbytes / nchunks < blosc2.proxy.BLOCK_MIN_CBYTES: - return None # Whether a dataset that reports a geometry is *served* from a file is # something only the answer to a range request can say: an HDF5 leaf or a # `.b2z` member reports one and is streamed all the same try: + # Nor is a frame of small chunks worth an index read: blosc2 declines + # to take a chunk below BLOCK_MIN_CBYTES apart, so nothing here would + # ever use a block, and the dataset keeps the behaviour it had before. + # Inside the `try`, since `api/info` need not carry what these read. + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) + if not nchunks or self.cbytes / nchunks < blosc2.proxy.BLOCK_MIN_CBYTES: + return None source = C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) source.adopt_index(self._pending_index) return source - except (_NotRanged, ValueError, NotImplementedError, RuntimeError, _httpx().HTTPError): - # Not ranged, not a contiguous frame, not an NDArray, or not - # reachable: whole chunks work for all of those + except _NotRanged as exc: + return _UNTRIED if exc.transient else None + except httpx.HTTPStatusError as exc: + # A busy or broken subscriber said nothing about how this is served + status = exc.response.status_code + return _UNTRIED if status >= 500 or status == 429 else None + except httpx.TransportError: + # Nothing was downloaded to find this out, so asking again is cheap + return _UNTRIED + except ( + _PartsMissing, + ValueError, + NotImplementedError, + RuntimeError, + KeyError, + IndexError, + struct.error, + httpx.HTTPError, + ): + # Not ranged, not a contiguous frame, not an NDArray, answered with + # something unreadable, or described by an `api/info` without the + # fields these read: whole chunks work for all of those return None def adopt_index(self, state) -> None: @@ -760,15 +843,15 @@ def max_ranges(self) -> int: def chunk_layout(self, nchunk: int): """Where the blocks of a chunk are; see :meth:`ByteRangeNDSource.chunk_layout`.""" - return self.block_source().chunk_layout(nchunk) + return self._ranged().chunk_layout(nchunk) def chunk_layouts(self, nchunks: Sequence[int]) -> list: """The same for several chunks; see :meth:`ByteRangeNDSource.chunk_layouts`.""" - return self.block_source().chunk_layouts(nchunks) + return self._ranged().chunk_layouts(nchunks) def block_plan(self, nchunk: int, nblocks: Sequence[int]) -> list[tuple[int, int, tuple]]: """The range reads covering *nblocks*; see :meth:`ByteRangeNDSource.block_plan`.""" - return self.block_source().block_plan(nchunk, nblocks) + return self._ranged().block_plan(nchunk, nblocks) def read_range(self, offset: int, size: int) -> bytes: """The bytes at [*offset*, *offset* + *size*) of the remote frame.""" diff --git a/src/blosc2/dict_store.py b/src/blosc2/dict_store.py index 035be5971..4435de09a 100644 --- a/src/blosc2/dict_store.py +++ b/src/blosc2/dict_store.py @@ -482,7 +482,11 @@ def member_window(self, key: str) -> tuple[int, int] | None: if relpath is None: return None window = self.offsets.get(relpath) - return (window["offset"], window["length"]) if window else None + if not window or not window.get("stored"): + # A deflated member is not a frame where it lies, whatever its suffix + # says, so there is no window onto it to hand out + return None + return (window["offset"], window["length"]) def _annotate_external_value( self, @@ -1000,7 +1004,14 @@ def _get_zip_offsets(self) -> dict[str, dict[str, int]]: filename_len = int.from_bytes(local_header[26:28], "little") extra_len = int.from_bytes(local_header[28:30], "little") data_offset = info.header_offset + 30 + filename_len + extra_len - self.offsets[info.filename] = {"offset": data_offset, "length": info.file_size} + # The *stored* length, which is what lies at `data_offset`: it is + # `file_size` only for a member kept whole, and a `.b2z` repacked + # by any other tool may well have deflated its members instead + self.offsets[info.filename] = { + "offset": data_offset, + "length": info.compress_size, + "stored": info.compress_type == zipfile.ZIP_STORED, + } return self.offsets def close(self) -> None: diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 2da6ad360..4f3c7e2a3 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -402,6 +402,8 @@ def __init__( # Blocks of the last few partly filled chunks, so a rewrite need not read # the chunk back out of the cache to find out what is already in it self._hot_payloads = {} + # The index as last written to the cache, to write it again only if it moves + self._saved_index = None nchunks = self._schunk_cache.nchunks nbits = nchunks * self._blocks_per_chunk self._fetched = bytearray((nbits + 7) // 8) if fresh else self._load_fetched(nchunks) @@ -412,6 +414,9 @@ def __init__( # Geometry alone cannot tell a replaced source from the one the cache was # filled from, so record whatever identity the source can name itself by stamp = getattr(self.src, "stamp", None) + # Read before writing, or the check below would compare the stamp with + # the copy of itself just written and pass for every writable cache + stored = None if fresh else self._schunk_cache.vlmeta.get("proxy-stamp") if stamp is not None and getattr(self._schunk_cache, "mode", None) != "r": # Not into a cache opened read-only, which `blosc2.open(path, mode="r")` # hands over for a persisted proxy: nothing may be written there, and a @@ -422,12 +427,7 @@ def __init__( # same remote bytes, checked here rather than taken on trust from how the # cache was come by: a `_cache=` handed in never passed `_reopen_cache`. adopt = getattr(self.src, "adopt_index", None) - if ( - adopt is not None - and not fresh - and stamp is not None - and self._schunk_cache.vlmeta.get("proxy-stamp") == stamp - ): + if adopt is not None and not fresh and stamp is not None and stored == stamp: adopt(self._schunk_cache.vlmeta.get("proxy-index")) if vlmeta: reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) @@ -585,7 +585,13 @@ def _save_fetched(self) -> None: # alone, which are the only ones a later fetch would ask about. state = getattr(self.src, "index_state", None) if state is not None and getattr(self.src, "stamp", None) is not None: - self._schunk_cache.vlmeta["proxy-index"] = state(self._partly_filled()) + index = state(self._partly_filled()) + # Only when it says something new: the offsets are the bulk of it and + # never change once read, so a slice-by-slice walk would otherwise + # rewrite eight bytes per chunk of the frame after every fetch + if index != self._saved_index: + self._schunk_cache.vlmeta["proxy-index"] = index + self._saved_index = index def _partly_filled(self) -> list[int]: """Chunks the cache holds some of the blocks of, but not all.""" @@ -803,9 +809,17 @@ def fetch(task): def _chunk_layouts(self, nchunks: list[int], max_concurrency: int | None): """Where the blocks of every one of *nchunks* are, in as few requests as fit.""" - if max(getattr(self.src, "max_ranges", 1), 1) > 1: - # The source batches the reads itself, so there is nothing to overlap - return self.src.chunk_layouts(nchunks) + batch = max(getattr(self.src, "max_ranges", 1), 1) + if batch > 1: + # The source reads a batch of them in one request, but only so many + # per request: the batches past the first are round trips like any + # other, so they are overlapped rather than run one after the next + tasks = list(batched(nchunks, batch)) + return [ + layout + for answers in self._run(self.src.chunk_layouts, tasks, max_concurrency) + for layout in answers + ] return self._run(self.src.chunk_layout, nchunks, max_concurrency) def _write_blocks(self, nchunk: int, payloads: dict[int, bytes], header: bytes) -> None: @@ -1375,7 +1389,9 @@ def index_state(self, keep: Sequence[int] = ()) -> dict: offsets = self._index[0] if self._index is not None else None return { "bpc": self.blocks_per_chunk, - "offsets": b"" if offsets is None else offsets.tobytes(), + # Little-endian whatever the host is: a cache directory outlives the + # machine that filled it, and a stamp cannot tell a byte order + "offsets": b"" if offsets is None else offsets.astype(" None: offsets = state.get("offsets") or b"" if offsets: nchunks = math.prod(math.ceil(s / c) for s, c in zip(self._shape, self._chunks, strict=True)) - array = np.frombuffer(offsets, dtype=np.int64) + array = np.frombuffer(offsets, dtype=" np.dtype: return self._dtype def get_chunk(self, nchunk: int) -> bytes: - offset = int(self._offsets[nchunk]) + # Read the index once: `_offsets` and `_extents` each take the lock, and + # what they hand back does not change after the first read + offsets, extents = self._frame_index() + offset = int(offsets[nchunk]) if offset < 0: return self._special_chunk(offset) - data = self.read_range(offset, int(self._extents[nchunk])) + data = self.read_range(offset, int(extents[nchunk])) return data[: struct.unpack(" bool: and an upper bound on the chunk's compressed size is already in hand from the frame's offsets. See the thresholds at the top of this module. """ - if int(self._offsets[nchunk]) < 0: + offsets, extents = self._frame_index() # once, rather than twice under the lock + if int(offsets[nchunk]) < 0: return False # a run-length chunk has no bytes in the file to skip if nwanted > self.blocks_per_chunk * BLOCK_MAX_FRACTION: return False - return int(self._extents[nchunk]) >= BLOCK_MIN_CBYTES + return int(extents[nchunk]) >= BLOCK_MIN_CBYTES def chunk_layout(self, nchunk: int) -> tuple[bytes, np.ndarray, np.ndarray] | None: """Read where the blocks of a chunk are: its header, bstarts and extents. @@ -1528,8 +1548,9 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: """ section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] + offsets = self._offsets if todo else () # once, not once per span below for batch in batched(todo, max(self.max_ranges, 1)): - spans = [(int(self._offsets[n]), section) for n in batch] + spans = [(int(offsets[n]), section) for n in batch] heads = self.read_ranges(spans) for nchunk, head in zip(batch, heads, strict=True): self._sections[nchunk] = head @@ -1539,6 +1560,11 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: def _parse_layout(self, head: bytes, section: int): """The layout a chunk's header section says it has, or None for no layout.""" nblocks = self.blocks_per_chunk + if len(head) < section: + # A chunk clipped by the end of the frame has no block-offsets section: + # asked for before the header is parsed, so a short one never reaches + # the fields below rather than tripping over their offsets + return None cbytes = struct.unpack("