Prune unread nested parquet leaves when a projected column is cast to a narrower type (nested schema pruning) - #23398
Closed
adriangb wants to merge 4 commits into
Closed
Prune unread nested parquet leaves when a projected column is cast to a narrower type (nested schema pruning)#23398adriangb wants to merge 4 commits into
adriangb wants to merge 4 commits into
Conversation
Registers the same wide list<struct> / struct parquet file with the
file's full schema and with a narrower declared schema, plus a
physically-narrow file as the floor. On main the narrow declared
schema reads exactly as many bytes as the full schema (the whole
column is fetched and clipped in memory by the adapter-inserted cast):
list_struct_narrow_schema: bytes_scanned=25.19 MB
list_struct_full_schema: bytes_scanned=25.19 MB
list_struct_physically_narrow: bytes_scanned=3.32 KB
select_events_narrow_schema 3.45 ms
select_events_full_schema 3.36 ms
select_events_physically_narrow 164 µs
Baseline for schema-driven nested projection pruning
(see apache/datafusion-comet#4859).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
Move-only: `ParquetReadPlan`, `StructFieldAccess`, `build_projection_read_plan` and the leaf-index/schema-pruning helpers move from row_filter.rs (2100+ lines) into projection_read_plan.rs. `PushdownChecker`/`PushdownColumns` become pub(crate) so the new module can keep using them. No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
…ast to a narrower type When a table's logical schema declares a nested column narrower than the physical parquet type (e.g. logical events: list<struct<x,y>> over a file with list<struct<x,y,+18 more>>), the expr adapter rewrites the projection into CAST(events AS narrow_type) — and the projection mask derivation expanded that column to every physical leaf, fetching and decoding subfields the cast then threw away. This adds schema-driven nested projection pruning (the equivalent of Spark's ParquetReadSupport.clipParquetSchema): when a projected root column is consumed only through such a cast, the leaf ProjectionMask is computed by walking the physical and cast-target type trees, matching struct fields by name, recursing through matching List/LargeList wrappers. Clipping keys off the cast target because that is exactly the contract of the runtime nested cast (cast_column consumes source children only by target-name lookup), so it applies to any PhysicalExprAdapter that emits whole-column casts. Safety/fallback rules (the clip is total; worst case = full read): - maps, dictionaries, wrapper-kind mismatches: keep all leaves (cast_column has no name-based Map arm) - struct level with zero name overlap: keep first child (>=1 leaf per level preserves struct definition levels, so `s IS NULL` semantics survive) - leaf type promotion stays the cast's job: emitted types keep the physical leaf types - unions with get_field accesses on the same root are supported; any whole-column reference wins Gated by datafusion.execution.parquet.nested_projection_pruning (default true). Comet reported reading 1.35 TB where Spark read 30.9 GB for the same pruned ReadSchema. On the parquet_nested_schema_pruning benchmark the narrow-schema scan drops from 25.19 MB (full column) to the pruned leaves only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
Same narrow-schema tables queried with
datafusion.execution.parquet.nested_projection_pruning=false, so the
before/after stays visible in one run:
select_events_narrow_schema 164 µs / 3.32 KB
select_events_narrow_schema_pruning_disabled 2.94 ms / 25.19 MB
select_events_physically_narrow (floor) 150 µs / 3.32 KB
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
github-actions
Bot
added
documentation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
mbutrovich
reviewed
Jul 22, 2026
mbutrovich
left a comment
mbutrovich
left a comment
Contributor
There was a problem hiding this comment.
The core approach here is the right one, and I think it's a better foundation than my #23391. Two things in particular:
- Keying the clip off a recognized
CastExpr(Column)node insidePushdownChecker(rather than offexpr.data_type()) makes the safety argument tight: the clip contract becomes exactly the runtime cast's contract. It also degrades per-root (unknown expressions force only their own root to a full read) instead of disabling the whole projection, and it composes withget_field. That's the generic shape #23391 can't reach without being rewritten. - The clip set (
Struct+List/LargeListonly;Map/FixedSizeListkept wholesale) is correct and matchescast_column's dispatch: maps and fixed-size lists fall through to Arrow's positionalcast_with_options(nested_struct.rs:202), so pruning them by name would be unsafe under field reordering. #23391 recurses into maps and is wrong for exactly this reason — good that this PR doesn't. - The leaf-count guard in
build_read_plan_with_cast_clipping(fall back to a full read when the arrow-embedded leaf count disagrees with the parquet schema) is the right safety net; #23391 has no equivalent.
I have two scope questions and one confirmation, inline. None of them are about the algorithm, which I think is solid; they're about how much surface the PR needs to carry.
| /// struct fields than the parquet file contains — the reader only | ||
| /// fetches and decodes the leaf columns the cast retains, instead of | ||
| /// the entire column | ||
| pub nested_projection_pruning: bool, default = true |
Contributor
There was a problem hiding this comment.
This flag is responsible for a large share of the PR's surface: the six proto-* files, parquet_writer.rs, file_formats.rs, information_schema.slt, configs.md, and the threading through source -> morselizer -> PreparedParquetOpen -> opener -> DecoderProjection. It's also what triggers the cargo-semver-checks major-version failure CI already reported (new pub field on the externally-constructible ParquetOptions).
The PR description itself notes there's "no scenario where a correctly-working clip should be disabled" and that the flag could be deprecated after a release or two. For a change that's semantically invisible (IO reduction only, worst case = today's behavior), a permanent serialized knob seems disproportionate.
Options, roughly in order of preference:
- Drop the flag entirely. The total-fallback design already makes this safe.
- If a bisection escape hatch is genuinely wanted, make it a non-serialized/session-only setting so it stays off the proto wire format and out of the semver surface.
Happy to be wrong if there's a concrete reason it must be plan-serialized, but as-is it roughly doubles the file count and forces a major-version bump for an optimization that's invisible to results.
| /// | ||
| /// Falls back to keeping all leaves of a root whenever its clipping is not | ||
| /// provably safe (see [`crate::nested_schema_pruning`]). | ||
| fn build_read_plan_with_cast_clipping( |
Contributor
There was a problem hiding this comment.
build_read_plan_with_cast_clipping + prune_type_by_kept_offsets union the leaves of a narrowing cast and get_field accesses on the same root. For the case this PR targets (Spark/Comet bake pruning into requiredSchema as a whole-column narrower type), the scan sees a pure CAST(col AS narrow) — no sibling get_field on that root. The get_field(CAST(col)) case (the SUM(s['x']) example in the description) is nested, and the visitor already handles it by clipping to the inner cast target, so it doesn't exercise the union path either.
Is there a concrete plan shape that produces a cast and a separate get_field on the same root at the scan boundary? If not, prune_type_by_kept_offsets and the union bookkeeping are speculative and could be deferred to a follow-up (with the simple per-root path covering the shipped feature), which would also shrink the review surface. If there is, a test that constructs that shape would document why the machinery exists.
| } | ||
| } | ||
| } | ||
| (DataType::List(p_item), DataType::List(t_item)) |
Contributor
There was a problem hiding this comment.
clip_type only recurses through List/LargeList; ListView, LargeListView, and Dictionary(value) fall to the _ arm and keep all leaves. That's safe, but cast_column does route those through the name-based recursive cast (nested_struct.rs:186-201), and requires_nested_struct_cast returns true for them (nested_struct.rs:477-483) — so they're clippable in principle and are just left on the table here.
Suggest a one-line comment noting this is a deliberate conservative choice (safe, matches "worst case = full read") and a candidate follow-up, so a future reader doesn't assume these shapes are unclippable.
|
|
||
| /// Fallback for a struct level with zero name overlap: keep the first child | ||
| /// that owns at least one leaf, skip the rest. | ||
| fn keep_first_child( |
Contributor
There was a problem hiding this comment.
The zero-name-overlap branch in clip_type (nested_schema_pruning.rs:116) exists to preserve one leaf when a struct level has no matching target field. But a cast with no field-name overlap is rejected at planning time: validate_struct_compatibility (nested_struct.rs:345) returns a plan error ("no field name overlap") before the reader is ever built. So for the DefaultPhysicalExprAdapter path this branch looks unreachable.
If it's kept for custom adapters that bypass that validation, worth a comment saying so; otherwise it and keep_first_child could be dropped in favor of the _ keep-all arm. Either way a targeted unit test (or a note on why it can't be tested) would clarify intent.
41 tasks
Contributor
|
Since this is still draft, has merge conflicts, and unaddressed feedback, I'm going to try to open a new PR branched off of main that incorporates the changes and suggested feedback and see if we can't still land this in DF 55.0. Thanks for the work thus far @adriangb! |
mbutrovich
added a commit
to mbutrovich/datafusion
that referenced
this pull request
Aug 4, 2026
Contributor
|
New PR: #24090 |
mithuncy
pushed a commit
to paradedb/datafusion
that referenced
this pull request
Aug 5, 2026
…arrower type (apache#24090) ## Which issue does this PR close? - Related to apache/datafusion-comet#4859. No DataFusion issue is filed for this one. ## Rationale for this change When a table declares a nested column narrower than the Parquet file's physical type, DataFusion reads every leaf of the column and drops the extra subfields in memory instead of skipping them at read time. - DefaultPhysicalExprAdapter rewrites the projected column into CAST(col AS narrow_type). - Projection mask derivation only understands literal get_field chains, so the cast's inner column expands to every physical leaf via ProjectionMask::roots. - Comet reported a production query reading 1.35 TB where plain Spark read 30.9 GB for the same pruned ReadSchema (datafusion-comet#4859). - Any embedder that hands DataFusion a pre-pruned schema (Comet, delta-rs, Iceberg integrations) hits the same gap. This is a port of apache#23398 onto current main. apache#23398 (stacked on the merged apache#23396 and apache#23397, superseding an earlier attempt at apache#23392) implements the fix and was reviewed favorably, but has merge conflicts against main since a follow-up refactor moved PushdownChecker and PushdownColumns into projection_read_plan.rs, and has four unanswered review comments. This PR reimplements the same approach against current main and resolves those four comments by construction: - Drops the nested_projection_pruning config flag; the clip's total fallback design makes a kill switch unnecessary. - Drops an unreachable zero-name-overlap code path; validate_struct_compatibility already rejects that case during physical planning. - Drops the untested union of a cast clip with a sibling get_field access on the same root, in favor of a full read fallback for that case. - Notes that ListView, LargeListView, and Dictionary wrappers are conservatively left unclipped, a candidate follow up. ## What changes are included in this PR? - New module datafusion/datasource-parquet/src/nested_schema_pruning.rs. clip_for_cast walks the physical and cast target type trees together, matches struct fields by name, recurses through List and LargeList, and returns the kept leaf offsets plus the pruned Arrow type in one pass. The clip is total: maps, dictionaries, wrapper-kind mismatches, and any shape it does not understand keep every leaf, so the worst case is today's full read. - PushdownChecker in projection_read_plan.rs now also collects CastColumnAccess entries for a CastExpr over a plain Column where the cast requires nested struct handling. Collection is opt in and only enabled for projection analysis, so filter pushdown is unchanged. - build_projection_read_plan routes to a new build_read_plan_with_cast_clipping when any cast access survives. It partitions referenced roots into whole column reads, cast clipped reads, get_field only reads, and full read fallbacks, including a fallback for a root reached by both a cast and a separate get_field access. - Fixed a pre-existing bug in the has_struct_columns fast path: it only tested the top level field type against Struct, so a LIST STRUCT root such as events was misclassified as containing no struct and skipped PushdownChecker entirely, meaning cast based clipping never fired for that shape. Replaced with a recursive contains_struct check through List, LargeList, ListView, LargeListView, FixedSizeList, Map, Dictionary, and RunEndEncoded. - No new configuration option. ## Are these changes tested? - 15 unit tests in nested_schema_pruning.rs: struct subset, reordering, leaf promotion, missing field null fill, nested struct in struct, list of struct, two levels of list of struct nesting, maps, dictionaries, wrapper mismatches, and an arrow-rs roundtrip test pinning that ProjectionMask::leaves over a subset of List Struct leaves emits exactly the predicted type. - 2 new unit tests in projection_read_plan.rs, plus the pre-existing struct preservation test, unchanged. - 8 integration tests in datafusion/core/tests/parquet/expr_adapter.rs, asserting identical results and a bytes_scanned drop of more than 2x: list of struct narrowing, top level struct, struct level nullability, get_field on a narrowed struct, mixed whole column and subfield access, filter pushdown enabled, a scan mixing a physically narrow and a wide file, and a regression test built from the ReadSchema and InputSchema shapes reported in datafusion-comet#4859 (a two level list STRUCT column with a dropped struct sibling, a dropped map sibling, and dropped top level columns). - New datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt covering the end to end SQL path. - The existing benchmark (already merged as part of apache#23397) had an assertion documenting the pre-fix baseline, narrow equals full. That assertion now fails as expected and has been flipped to assert narrow reads less than half of full. Measured for the top level struct case: narrow_schema=1081 bytes vs full_schema=8398289 bytes, matching the physically_narrow floor of 1081 bytes. ## Are there any user-facing changes? No API changes and no new configuration option. Behavior is IO reduction only, results are unchanged. --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #23396 (move-only refactor) and #23397 (benchmark). Until those merge, this PR shows their commits too; the feature itself is the last two commits (
feat: prune unread nested parquet leaves...andbench: add pruning-disabled variants...), ~900 hand-written lines of which roughly half are tests. Supersedes #23392.Which issue does this PR close?
Rationale for this change
When a table's logical file schema declares a nested column narrower than the physical parquet type — e.g. the table declares
events: LIST<STRUCT<x, y>>over a file containingevents: LIST<STRUCT<x, y, +16 more fields>>— DataFusion reads every leaf of the column and drops the extra subfields in memory:DefaultPhysicalExprAdapterrewrites the projected column intoCAST(events AS narrow_type).get_fieldchains; the cast's inner column falls into theProjectionMask::rootspath, expanding to all physical leaves.nested_struct::cast_column) discards the unrequested subfields.Engines like Spark communicate nested projection pruning to the scan exactly this way — as a clipped read schema, not as
get_fieldexpressions (Spark'sSchemaPruningrule consumes the field-access expressions and bakes the result into the scan'srequiredSchema). Comet measured a production query reading 1.35 TB where plain Spark read 30.9 GB for the same prunedReadSchema. The same behavior is reproducible in pure DataFusion by declaring a narrower nested type inCREATE EXTERNAL TABLEand watchingbytes_scanned. Any embedder that hands DataFusion a pre-pruned schema (Comet, delta-rs, Iceberg integrations) benefits.This PR implements the equivalent of Spark's
ParquetReadSupport.clipParquetSchema: when a projected root column is consumed only through such a cast, the leafProjectionMaskis computed by walking the physical and cast-target type trees, matching struct fields by name, so only the leaves the cast retains are fetched and decoded. The adapter-inserted cast then degrades to a cheap reorder/rename/null-fill/promotion.Why key off the cast target instead of the logical-vs-physical schema diff? The cast target is exactly the contract of the runtime nested cast:
cast_columnconsumes source struct children only by looking up target field names, recursively through list wrappers, so physical subtrees the target never names are provably dead. It also works for customPhysicalExprAdapters (which are allowed to inject references to physical-only columns — schema-driven clipping could under-read there, expression-driven clipping degrades safely to a full read), and it requires no opener signature changes. A bareColumnwhose logical type is narrower than the physical file is not a working configuration today, so threading the logical schema would add no coverage.Benchmark results (see #23397; wide
list<struct>file, narrow declared schema):The
SUM(s['x'])shape on a schema-evolved struct — whereget_field(CAST(col))currently defeats the existing expression-level leaf clipping entirely (the bottom-up adapter rewrite meansPushdownCheckerno longer sees a plainColumnunderget_field) — improves ~80–90% as well, since it now clips to the cast target.What changes are included in this PR?
On top of the stacked #23396/#23397:
datafusion/datasource-parquet/src/nested_schema_pruning.rs:clip_for_castwalks physical vs cast-target types (name-matched struct children, case-sensitive to matchcast_column; recurses through matchingList/LargeListwrappers) and returns kept leaf offsets;prune_type_by_kept_offsetsderives the Arrow type the reader emits for a kept-offset union. The clip is total — maps (no name-based Map arm incast_column), dictionaries, wrapper-kind mismatches, and zero-name-overlap struct levels keep all their leaves, so the worst case is today's full read. Every struct level keeps ≥1 leaf, so struct definition levels (ands IS NULLsemantics) survive.PushdownCheckeroptionally collectsCastColumnAccess(aCastExprover a plainColumnwhererequires_nested_struct_cast(physical, cast_type)), enabled only for projection analysis — the row-filter path is mechanically untouched.build_projection_read_planunions cast-clipped leaves with existingget_fieldleaf accesses per root; any whole-column reference wins; roots with onlyget_fieldaccesses keep the bit-identical legacy path. A defensive leaf-count check falls back to a whole-root read if the arrow/parquet leaf mapping ever disagrees.datafusion.execution.parquet.nested_projection_pruning(defaulttrue), threaded source → morselizer → opener →DecoderProjection, plus proto field + docs +information_schemaentries. The flag exists purely as an escape hatch / bisection aid for a change in the IO path — there is no scenario where a correctly-working clip should be disabled — and could be deprecated after a release or two.Are these changes tested?
Yes:
ProjectionMask::leavesover a subset ofList<Struct>leaves emits exactly the predicted type and preserves null list rows / null struct elements.datafusion/core/tests/parquet/expr_adapter.rs, each asserting identical results with the flag on and off and thatbytes_scanneddrops by more than 2×: list-of-struct narrowing (subset + reorder +Int32→Int64leaf promotion + missing-subfield null-fill), top-level struct, struct-level nullability (s IS NULL),get_fieldon a narrowed struct, mixed whole-column + subfield access, filter pushdown enabled, and a scan mixing a physically-narrow and a wide file.datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.sltcovering the end-to-end SQL path (CREATE EXTERNAL TABLEwith a narrower nested type) with the setting toggled both ways.Are there any user-facing changes?
A new configuration option,
datafusion.execution.parquet.nested_projection_pruning(defaulttrue), documented inconfigs.md. No API changes; behavior is semantically invisible (IO reduction only).Deliberately out of scope, noted for follow-ups:
Maparm innested_struct::cast_column).get_fieldpaths with cast targets (naively unsafe: null-filling a non-nullable logical sibling changes per-batch struct-compatibility validation outcomes).🤖 Generated with Claude Code
https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd