IN LIST: reuse primitive filters for FixedSizeBinary - #24102
Draft
geoffreyclaude wants to merge 3 commits into
Draft
IN LIST: reuse primitive filters for FixedSizeBinary#24102geoffreyclaude wants to merge 3 commits into
geoffreyclaude wants to merge 3 commits into
Conversation
geoffreyclaude
force-pushed
the
codex/in-list-fixed-size-binary-hashset
branch
3 times, most recently
from
August 5, 2026 12:00
9915d25 to
6f820d7
Compare
Contributor
Author
|
run benchmark in_list |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-fixed-size-binary-hashset (6f820d7) to b6cccfe diff Run configurationrun benchmark in_list
baseline:
ref: "b6cccfeef1a78ab424ffec87afb40958c33fb897"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-fixed-size-binary-hashset (6f820d7) to b6cccfe diff Run configurationrun benchmark in_list
baseline:
ref: "b6cccfeef1a78ab424ffec87afb40958c33fb897"CPU Details (lscpu)Details
Resource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-fixed-size-binary-hashset (6f820d7) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-fixed-size-binary-hashset (6f820d7) to b6cccfe diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "b6cccfeef1a78ab424ffec87afb40958c33fb897"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-fixed-size-binary-hashset (6f820d7) to 426b351 (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-fixed-size-binary-hashset (6f820d7) to b6cccfe diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "b6cccfeef1a78ab424ffec87afb40958c33fb897"CPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24102 +/- ##
==========================================
+ Coverage 80.91% 81.06% +0.14%
==========================================
Files 1102 1107 +5
Lines 377102 382143 +5041
Branches 377102 382143 +5041
==========================================
+ Hits 305143 309780 +4637
- Misses 53769 54056 +287
- Partials 18190 18307 +117 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
codex/in-list-fixed-size-binary-hashset
branch
2 times, most recently
from
August 6, 2026 17:27
0ae4133 to
89d1595
Compare
This was referenced Aug 8, 2026
geoffreyclaude
force-pushed
the
codex/in-list-fixed-size-binary-hashset
branch
3 times, most recently
from
August 9, 2026 07:46
d349e82 to
59f116f
Compare
geoffreyclaude
force-pushed
the
codex/in-list-fixed-size-binary-hashset
branch
from
August 9, 2026 08:09
59f116f to
59667db
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
A
FixedSizeBinary(N)array stores every value as exactlyNbytes. For widths that match primitive representations already optimized by theIN LISTstack, those bytes can be used as an exact membership key:UInt8UInt16UInt32HashSetUInt64HashSetDecimal128/i128HashSetThis is only an internal representation change. The bytes are never interpreted arithmetically:
Decimal128, for example, is just a convenient 128-bit carrier. Because the list and input are converted in exactly the same way, two primitive keys are equal if and only if the original fixed-size byte strings are equal. The mapping therefore preserves equality regardless of machine byte order.This avoids the generic Arrow comparison path while reusing the simple strategies already selected by representation and list size:
Arrow buffers are normally aligned well enough to expose the same bytes as primitive values without copying. Arrow also permits valid arrays backed by unaligned buffers, so the adapter checks alignment and copies into aligned primitive storage only in that uncommon case.
Widths other than 1, 2, 4, 8, and 16 bytes continue to use the existing generic filter.
What changes are included in this PR?
FixedSizeBinaryadapter that exposes supported widths through same-sized primitive representations.FixedSizeBinary(N)type as the list.IN,NOT IN, input nulls, and nulls in the list through the shared static-filter result path.Are these changes tested?
Yes. Focused coverage includes:
INlist;INandNOT IN;Are there any user-facing changes?
No. This is an internal performance optimization only.
Local benchmark snapshot
Built and run in separate target directories after compilation completed, using identical benchmark source:
Criterion defaults were used, comparing median point estimates. Filter construction is outside the timed loop. Lower is better; changes within +/-5% are treated as noise.
Compared baselines: #23014 -> #24102
The 12 rows cover a 1-byte branchless filter, a 2-byte bitmap filter, and 16-byte branchless and hash-set filters. All improved, with a 71.5% geometric-mean time reduction. By filter, the reductions were 83.5% for branchless, 91.2% for bitmap, and 39.4% for hash-set rows.
fixed_size_binary/fsb1/list=16/match=0%fixed_size_binary/fsb1/list=16/match=50%fixed_size_binary/fsb2/list=64/match=0%fixed_size_binary/fsb2/list=64/match=50%fixed_size_binary/fsb16/list=4/match=0%fixed_size_binary/fsb16/list=4/match=50%fixed_size_binary/fsb16/list=64/match=0%fixed_size_binary/fsb16/list=64/match=50%fixed_size_binary/fsb16/list=256/match=0%fixed_size_binary/fsb16/list=256/match=50%fixed_size_binary/fsb16/list=10000/match=0%fixed_size_binary/fsb16/list=10000/match=50%