Skip to content

feat: preserve bitmap-backed RowSelection through ParquetAccessPlan - #24186

Open
haohuaijin wants to merge 4 commits into
apache:mainfrom
haohuaijin:preserve-bitmap-row-selection
Open

feat: preserve bitmap-backed RowSelection through ParquetAccessPlan#24186
haohuaijin wants to merge 4 commits into
apache:mainfrom
haohuaijin:preserve-bitmap-row-selection

Conversation

@haohuaijin

@haohuaijin haohuaijin commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Parquet 59 added a bitmap (BooleanBuffer) backing for RowSelection alongside the RLE selector form. However, ParquetAccessPlan always materialized selectors, so a caller-provided bitmap selection (via the ParquetRowSelection extension or an external ParquetAccessPlan) was converted to RLE before reaching the parquet reader. For highly fragmented selections the bitmap form is significantly cheaper to build, slice, and intersect, so it should survive the trip through the access plan.

What changes are included in this PR?

  • ParquetAccessPlan::try_new_from_overall_row_selection splits a mask-backed selection per row group with RowSelection::split_off, which slices the BooleanBuffer (O(bitmap) overall) instead of materializing selectors. The selector path is unchanged.
  • ParquetAccessPlan::into_overall_row_selection produces a bitmap-backed overall selection when any row group selection is bitmap-backed, promoting selector-backed groups (e.g. groups that page index pruning intersected back to selectors). When no group is bitmap-backed, the existing selector path is preserved.
  • ParquetAccessPlan::scan_selection promotes the incoming selection to a bitmap when the existing selection is bitmap-backed, since RowSelection::intersection only stays mask-backed when both sides are masks; the intersection is then a bitwise AND instead of a selector merge.
  • reverse_row_selection (reverse scans for TopK) slices the bitmap per scanned row group and re-concatenates in reverse, keeping the mask backing; a debug_assert guards the "selection covers exactly the scanned row groups" contract.
  • Row-count validation now uses row_count() + skipped_row_count(), which avoids materializing selectors from a mask just to count rows.

Are these changes tested?

Yes:

  • Unit tests for the mask path of try_new_from_overall_row_selection (round trip + invalid row count), mixed-backing promotion in into_overall_row_selection, mask preservation in scan_selection, and bitmap preservation through PreparedAccessPlan::reverse (including a fully scanned row group).
  • End-to-end tests in external_access_plan.rs: a bitmap ParquetRowSelection spanning row groups, and a bitmap selection combined with a predicate that prunes a row group via statistics.
  • Existing selector-based tests are unchanged and passing (datafusion-datasource-parquet unit tests and the parquet_integration suite).

Are there any user-facing changes?

No API changes. Behavior change: when any per-row-group selection is bitmap-backed, the overall RowSelection handed to the parquet reader is now bitmap-backed instead of selector-backed (semantically identical selection).

Parquet 59 added a bitmap (BooleanBuffer) backing for RowSelection in
addition to the RLE selector form. Previously ParquetAccessPlan always
materialized selectors, so a caller-provided bitmap selection was
converted to RLE before reaching the parquet reader, which is wasteful
for fragmented selections.

This change keeps bitmap-backed selections bitmap-backed end to end:

* `try_new_from_overall_row_selection` splits a mask-backed selection
  per row group with `split_off`, which slices the BooleanBuffer
  instead of materializing selectors.
* `into_overall_row_selection` builds a bitmap-backed overall selection
  when any row group selection is bitmap-backed, promoting
  selector-backed groups (e.g. ones page index pruning intersected).
* `scan_selection` promotes the incoming selection to a bitmap when
  intersecting with an existing bitmap-backed selection, so the
  intersection is a bitwise AND and stays mask-backed.
* `reverse_row_selection` slices and re-concatenates the bitmap for
  reverse scans, with a debug_assert guarding the row-count contract.
@github-actions github-actions Bot added core Core DataFusion crate datasource Changes to the datasource crate labels Aug 8, 2026
@codecov-commenter

codecov-commenter commented Aug 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.07%. Comparing base (0ef844e) to head (605d2fb).
⚠️ Report is 20 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24186      +/-   ##
==========================================
+ Coverage   81.02%   81.07%   +0.04%     
==========================================
  Files        1105     1106       +1     
  Lines      380671   382067    +1396     
  Branches   380671   382067    +1396     
==========================================
+ Hits       308436   309756    +1320     
- Misses      54002    54038      +36     
- Partials    18233    18273      +40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@haohuaijin
haohuaijin marked this pull request as ready for review August 9, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve bitmap-backed RowSelection in ParquetAccessPlan

2 participants