Part of #24119.
Background
#24090 added datafusion/datasource-parquet/src/nested_schema_pruning.rs, which clips a nested column's Parquet leaves to the subset a narrowing cast actually consumes. clip_type recurses through Struct, List and LargeList, and keeps every leaf for anything else:
// Anything else, leaf pairs, wrapper-kind mismatches, maps,
// dictionaries, fixed-size lists, views, is kept wholesale.
_ => keep_all_leaves(physical, next_leaf, kept),
The module doc records this as deliberate:
Nor are ListView/LargeListView/Dictionary wrappers clipped here, even though cast_column does recurse through them by name. That is a conservative choice (safe, since the worst case is still just a full read) left as a candidate follow-up rather than something this module currently handles.
So a column typed ListView<Struct<a, b, c>> or Dictionary<_, Struct<a, b, c>> still reads all three leaves under CAST(col AS ListView<Struct<a>>), where the equivalent List<Struct<...>> reads one.
What this issue asks for
Extend clip_type to recurse through the wrapper kinds where the runtime cast provably consumes children by name:
ListView / LargeListView — cast_column recurses through them the same way it does List / LargeListFixedSizeListDictionary (clip the value type)RunEndEncoded (clip the value type)
nested_child in the same module already enumerates exactly this set for count_leaves / contains_struct, so the leaf accounting side is in place; what's missing is the matching arms in clip_type that rebuild the wrapper around the pruned child type.
Explicitly out of scope: Map. Map values are not clippable — the runtime cast routes maps through Arrow's positional struct cast, which requires all children to be present. That must stay a full read.
Requirements
- Preserve the existing invariant that every clipped struct level keeps at least one leaf (the reader reconstructs ancestor validity from the definition levels of surviving leaves), and the
unclippable bail-out when it cannot be met.
- Preserve the total-fallback property: an unrecognised shape keeps all leaves, so the worst case stays today's full read.
- Wrapper-kind mismatches between physical and target (e.g. physical
List, target ListView) must keep falling through to a full read.
Test coverage to add
Mirroring the existing unit tests in nested_schema_pruning.rs:
ListView<Struct<..>> and LargeListView<Struct<..>> narrowed by a castFixedSizeList<Struct<..>>Dictionary<_, Struct<..>> and RunEndEncoded<_, Struct<..>>- wrapper-kind mismatch → no clipping
Map → still no clipping (guard against a regression that starts clipping map values)- an arrow-rs roundtrip test pinning that
ProjectionMask::leaves over the predicted leaf subset emits exactly the predicted type, as the List<Struct> test does today
Notes
These wrapper types rarely appear as Parquet-derived Arrow types in practice, so this is a completeness item rather than a hot path. Filing so the deliberate gap is tracked.
Part of #24119.
Background
#24090 added
datafusion/datasource-parquet/src/nested_schema_pruning.rs, which clips a nested column's Parquet leaves to the subset a narrowing cast actually consumes.clip_typerecurses throughStruct,ListandLargeList, and keeps every leaf for anything else:The module doc records this as deliberate:
So a column typed
ListView<Struct<a, b, c>>orDictionary<_, Struct<a, b, c>>still reads all three leaves underCAST(col AS ListView<Struct<a>>), where the equivalentList<Struct<...>>reads one.What this issue asks for
Extend
clip_typeto recurse through the wrapper kinds where the runtime cast provably consumes children by name:ListView/LargeListView—cast_columnrecurses through them the same way it doesList/LargeListFixedSizeListDictionary(clip the value type)RunEndEncoded(clip the value type)nested_childin the same module already enumerates exactly this set forcount_leaves/contains_struct, so the leaf accounting side is in place; what's missing is the matching arms inclip_typethat rebuild the wrapper around the pruned child type.Explicitly out of scope:
Map. Map values are not clippable — the runtime cast routes maps through Arrow's positional struct cast, which requires all children to be present. That must stay a full read.Requirements
unclippablebail-out when it cannot be met.List, targetListView) must keep falling through to a full read.Test coverage to add
Mirroring the existing unit tests in
nested_schema_pruning.rs:ListView<Struct<..>>andLargeListView<Struct<..>>narrowed by a castFixedSizeList<Struct<..>>Dictionary<_, Struct<..>>andRunEndEncoded<_, Struct<..>>Map→ still no clipping (guard against a regression that starts clipping map values)ProjectionMask::leavesover the predicted leaf subset emits exactly the predicted type, as theList<Struct>test does todayNotes
These wrapper types rarely appear as Parquet-derived Arrow types in practice, so this is a completeness item rather than a hot path. Filing so the deliberate gap is tracked.