Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
9dc41b8
Add a plan for reading Blosc2 containers through fsspec URLs
FrancescAlted Aug 16, 2026
1ee2f32
Read and write Blosc2 containers through fsspec URLs
FrancescAlted Aug 16, 2026
2a5dd2c
Open remote containers through a local fsspec cache
FrancescAlted Aug 16, 2026
afbdcea
Read remote frames chunk by chunk with lazy=True
FrancescAlted Aug 16, 2026
f5b4ba2
Name the http test after what it checks
FrancescAlted Aug 16, 2026
9ba346e
Let a Proxy pick up the cache left by an earlier run
FrancescAlted Aug 16, 2026
c5961c5
Keep the install page about installing
FrancescAlted Aug 16, 2026
ea7543b
Trim the fsspec docs to one home per fact
FrancescAlted Aug 16, 2026
7e6d6e6
Let lazy= and cache_storage= compose
FrancescAlted Aug 16, 2026
01ca574
Stamp the chunk cache with fsspec's ukey, not guessed metadata
FrancescAlted Aug 16, 2026
950d3bf
Save a container to an fsspec URL in one PUT
FrancescAlted Aug 16, 2026
2da443f
Add an fsspec read/write example
FrancescAlted Aug 16, 2026
f305665
Point the fsspec docs at the runnable example
FrancescAlted Aug 16, 2026
215b6b7
Fetch a remote chunk in one range read, statelessly
FrancescAlted Aug 16, 2026
77f4304
Overlap chunk fetches in Proxy.fetch with a thread pool
FrancescAlted Aug 16, 2026
5beb7ef
Default a lazy fsspec proxy to 8 concurrent fetches
FrancescAlted Aug 16, 2026
be00ee8
Add a concurrent-fetch example for lazy fsspec arrays
FrancescAlted Aug 16, 2026
8e4364c
Link the concurrency example from the docs
FrancescAlted Aug 16, 2026
c7f916d
Bring the plan up to date with what phase 3 became
FrancescAlted Aug 16, 2026
1fd52c2
Install fsspec in the test group so CI actually runs its tests
FrancescAlted Aug 16, 2026
2cc1f12
Add a benchmark for the concurrency default against a real endpoint
FrancescAlted Aug 16, 2026
090e149
Document the moto[server] recipe in the benchmark
FrancescAlted Aug 16, 2026
01dfef5
Fix aget_chunk against real async filesystems
FrancescAlted Aug 16, 2026
42fac93
Test the fsspec support against a real S3 endpoint
FrancescAlted Aug 16, 2026
d2fe249
Address the PR review: file:// URLs, cache identity, cache geometry
FrancescAlted Aug 16, 2026
ea09cb8
Fix six defects found reviewing the frame reader
FrancescAlted Aug 16, 2026
5aec265
Fix what two reviews found in the fsspec support
FrancescAlted Aug 16, 2026
bf87015
Check the source identity when a Proxy adopts a cache
FrancescAlted Aug 16, 2026
ff47bae
Track fetched chunks explicitly in the Proxy cache
FrancescAlted Aug 16, 2026
2b3a112
Fix file:// drive URLs on POSIX and .b2d detection with a query
FrancescAlted Aug 16, 2026
82a971a
Trim the fsspec proxy plumbing
FrancescAlted Aug 16, 2026
c6285e1
Snapshot the sidecar caches before iterating them
FrancescAlted Aug 16, 2026
473511f
Fetch blocks, not whole chunks, from a lazy fsspec array
FrancescAlted Aug 16, 2026
449d459
Say what lazy=True fetches now that it is not whole chunks
FrancescAlted Aug 16, 2026
aacd9cc
Stop truncating lazy chunks down to their header
FrancescAlted Aug 16, 2026
6ff74e2
Analyse and measure block-granular fsspec downloads
FrancescAlted Aug 16, 2026
4e1db6c
Benchmark block fetching end to end, against a moto S3 server
FrancescAlted Aug 16, 2026
d434ad2
Record what route A became
FrancescAlted Aug 16, 2026
1f569f0
Note the rewrite-per-fetch ceiling where it lives
FrancescAlted Aug 17, 2026
66f9bfe
Stop overselling persisted block layouts
FrancescAlted Aug 17, 2026
cbe577e
Stop reading a chunk back to find out what is already in it
FrancescAlted Aug 17, 2026
91d5e75
Record what the route B prototype settled
FrancescAlted Aug 17, 2026
161b008
Finish waiting for modal screens instead of pausing once
FrancescAlted Aug 17, 2026
372dc28
Address the review: keep the proxy's bookkeeping its own
FrancescAlted Aug 17, 2026
dbf6cdf
Address the review: chunks that cannot be spliced, and evictions that…
FrancescAlted Aug 17, 2026
484f502
Do not read block offsets out of a chunk that has none
FrancescAlted Aug 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions RELEASE_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,48 @@ XXX version-specific blurb XXX

### Improvements

* New `blosc2[fsspec]` extra: `blosc2.open()`, `save_array()` and `save_tensor()`
accept any [fsspec](https://filesystem-spec.readthedocs.io) URL — `s3://`,
`gs://`, `zip://`, chained ones like `zip://inner.b2nd::s3://bucket/a.zip`.
`open()` reads the container whole, or through a staleness-checked local copy
with `cache_storage=` (which is what covers `.b2d` stores, sparse frames,
`offset` and `mmap_mode`), or a piece at a time with `lazy=True`, which
leaves a huge frame where it is and fetches only what a slice touches
through the new `blosc2.FsspecNDSource`. The two combine: `lazy=True` with a
`cache_storage=` keeps what was fetched there, so a later run starts from
it. Protocol drivers (`s3fs`, `gcsfs`...) and credentials stay the caller's
business. On the write side `NDArray.save()` and `blosc2.save()` upload the
whole array as one object; containers cannot be *backed* by a URL while they
are written (the C layer rewrites a frame's header and offsets as chunks land,
which an object store has no way to serve), so constructors given a URL now say
that instead of failing deep in C.

* `Proxy.fetch()` takes a `max_concurrency=` argument, and reads it from the
source when the source has one, so `blosc2.open(url, lazy=True,
max_concurrency=...)` overlaps its chunk fetches in a thread pool. Ordinary
slicing benefits, not just the async `afetch()`. A lazy fsspec proxy defaults
to 8, matching what `afetch()` already used for remote sources; pass 1 for a
protocol with no latency to hide, where the pool costs ~10 µs per chunk and
saves nothing. Other sources stay serial unless asked, since this is only safe
for a thread-safe `get_chunk`.

* `blosc2.open(url, lazy=True)` fetches **blocks** rather than whole chunks when
a slice lands in a small part of a large one. With the partitions
`blosc2.asarray()` picks by default a chunk holds a hundred-odd blocks, so a
point or window read costs about 1% of what it used to: measured against S3,
5-17x faster on arrays with multi-megabyte chunks and 2-5x on 1 MB ones. It is
never a loss, because the two thresholds that decide it — a chunk under a
megabyte is one cheap request anyway, and wanting more than half of a chunk's
blocks is wanting the chunk — need nothing read to answer. Peak memory drops
with the traffic, since a fetch in flight is now a block rather than a chunk.
`bench/ndarray/fsspec-block-granularity.py` measures both on any array.

* `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an
earlier run instead of failing on the existing file, so a proxy's cache can
outlive the process. The cache must come from a proxy over a source of the same
shape and dtype; anything else at that path raises. A cache that holds only
some blocks of a chunk keeps them across runs too.

* Querying a `utf8()` column through its FULL index no longer materializes the
index vocabulary. The query literal is turned into an alphabetical rank by
bisecting the vocabulary sidecar instead, so a lookup reads a few blocks
Expand Down
Loading
Loading