Skip to content

Devalue uxarray to make it less expensive#1588

Open
cmdupuis3 wants to merge 13 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/devalue
Open

Devalue uxarray to make it less expensive#1588
cmdupuis3 wants to merge 13 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/devalue

Conversation

@cmdupuis3

@cmdupuis3 cmdupuis3 commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Partly addresses #1583

Overview

This PR is to resolve suboptimal usage of .values throughout the repo, primarily by deferring to lazy xarray and dask operations. See the table of usage sites in Issue #1583.

This specific PR is for the first two tables of usage sites; dead code and bugfixes aren't really included here. Aside from simple cases, some cases were solved by branching on whether the data type was already chunked, so scalability for some parts will depend on if you're using numpy or xarray/dask arrays at those points.

PR Checklist

General

  • An issue is linked created and linked
  • Add appropriate labels
  • Filled out Overview and Expected Usage (if applicable) sections

Testing

  • Adequate tests are created if there is new functionality
  • Tests cover all possible logical paths in your function
  • Tests are not too basic (such as simply calling a function and nothing else)

Documentation

  • Docstrings have been added to all new functions
  • Docstrings have updated with any function changes

@cmdupuis3 cmdupuis3 self-assigned this Jul 18, 2026
@cmdupuis3 cmdupuis3 added scalability Related to scalability & performance efforts run-benchmark Run ASV benchmark workflow labels Jul 18, 2026
@github-actions

github-actions Bot commented Jul 18, 2026

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [ac309b0] After [42d9bdc] Ratio Benchmark (Parameter)
- 524M 336M 0.64 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
- 644M 335M 0.52 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
- 704±5ms 573±3ms 0.81 mpas_ocean.CrossSections.time_const_lat('120km', 1)
- 357±10ms 288±1ms 0.81 mpas_ocean.CrossSections.time_const_lat('120km', 2)
- 183±1ms 149±0.7ms 0.82 mpas_ocean.CrossSections.time_const_lat('120km', 4)
- 450M 328M 0.73 mpas_ocean.Gradient.peakmem_gradient('480km')

Benchmarks that have stayed the same:

Change Before [ac309b0] After [42d9bdc] Ratio Benchmark (Parameter)
10.7±0.1μs 11.0±0.2μs 1.03 bench_connectivity.Connectivity.time_edge_face('120km')
11.1±0.09μs 11.2±0.1μs 1 bench_connectivity.Connectivity.time_edge_face('480km')
10.5±0.08μs 10.9±0.09μs 1.03 bench_connectivity.Connectivity.time_edge_node('120km')
11.2±0.1μs 11.2±0.1μs 1 bench_connectivity.Connectivity.time_edge_node('480km')
10.7±0.1μs 10.8±0.09μs 1.01 bench_connectivity.Connectivity.time_face_edge('120km')
11.1±0.08μs 11.1±0.2μs 1 bench_connectivity.Connectivity.time_face_edge('480km')
10.9±0.08μs 10.9±0.2μs 1 bench_connectivity.Connectivity.time_face_face('120km')
11.2±0.1μs 11.1±0.08μs 1 bench_connectivity.Connectivity.time_face_face('480km')
22.3±0.1μs 21.4±0.3μs 0.96 bench_connectivity.Connectivity.time_face_node('120km')
22.7±0.07μs 22.5±0.2μs 0.99 bench_connectivity.Connectivity.time_face_node('480km')
10.9±0.08μs 10.8±0.1μs 0.99 bench_connectivity.Connectivity.time_node_edge('120km')
11.0±0.03μs 11.3±0.08μs 1.02 bench_connectivity.Connectivity.time_node_edge('480km')
10.7±0.1μs 11.0±0.07μs 1.03 bench_connectivity.Connectivity.time_node_face('120km')
11.1±0.1μs 11.1±0.3μs 1 bench_connectivity.Connectivity.time_node_face('480km')
334M 334M 1 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
365M 363M 0.99 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
14.6±0.05ms 14.3±0.03ms 0.98 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
3.76±0.08ms 3.68±0.02ms 0.98 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
19.0±0.1ms 18.9±0.04ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
2.18±0.02ms 2.18±0.01ms 1 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
571±10ms 566±1ms 0.99 import.Imports.timeraw_import_uxarray
926±10ns 940±20ns 1.02 mpas_ocean.CheckNorm.time_check_norm('120km')
895±10ns 884±20ns 0.99 mpas_ocean.CheckNorm.time_check_norm('480km')
837±10ms 830±10ms 0.99 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
53.7±0.4ms 53.7±0.5ms 1 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
676±20μs 678±5μs 1 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
588±10μs 585±20μs 0.99 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
5.46±0.04ms 5.39±0.03ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.96±0.1ms 3.92±0.03ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
3.45±0.01s 3.45±0.02s 1 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
225±1ms 226±5ms 1 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
18.1±0.04ms 18.1±0.02ms 1 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
1.04±0.01ms 1.03±0.01ms 0.98 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
10.7±0.1ms 10.6±0.01ms 0.99 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
715±30μs 725±10μs 1.01 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
548±3ms 523±2ms 0.95 mpas_ocean.CrossSections.time_const_lat('480km', 1)
277±3ms 262±1ms 0.94 mpas_ocean.CrossSections.time_const_lat('480km', 2)
142±2ms 135±0.4ms 0.95 mpas_ocean.CrossSections.time_const_lat('480km', 4)
24.9±0.7ms 24.3±0.2ms 0.98 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.35±0.1ms 3.09±0.05ms 0.92 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
959±4ms 950±2ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False)
53.0±2ms 53.2±1ms 1 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True)
84.1±1ms 84.6±0.4ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False)
5.64±0.3ms 5.56±0.1ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True)
349M 349M 1 mpas_ocean.Gradient.peakmem_gradient('120km')
174±1ms 173±0.3ms 1 mpas_ocean.Gradient.time_gradient('120km')
12.5±0.07ms 12.2±0.02ms 0.98 mpas_ocean.Gradient.time_gradient('480km')
224±1μs 224±0.4μs 1 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
133±0.6μs 131±0.5μs 0.99 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
298M 298M 1 mpas_ocean.Integrate.peakmem_integrate('120km')
277M 277M 1 mpas_ocean.Integrate.peakmem_integrate('480km')
219±3μs 215±1μs 0.98 mpas_ocean.Integrate.time_integrate('120km')
203±0.4μs 196±2μs 0.97 mpas_ocean.Integrate.time_integrate('480km')
183±1ms 185±2ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
183±3ms 185±2ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
181±1ms 186±0.6ms 1.03 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
13.8±0.3ms 13.9±0.09ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
13.6±0.4ms 13.8±0.2ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
13.8±0.3ms 13.7±0.09ms 1 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
352±6μs 356±1μs 1.01 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
357±3μs 359±4μs 1.01 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
334±2μs 332±1μs 0.99 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
340±2μs 334±2μs 0.98 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
241±3ms 242±0.6ms 1.01 mpas_ocean.RemapDownsample.time_bilinear_remapping
286±2ms 291±5ms 1.02 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
4.76±0.04ms 4.74±0.03ms 1 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
1.42±0s 1.43±0.01s 1.01 mpas_ocean.RemapUpsample.time_bilinear_remapping
36.5±0.5ms 36.6±0.3ms 1 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
9.52±0.1ms 9.60±0.1ms 1.01 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
28.8±0.08ms 29.1±1ms 1.01 mpas_ocean.ZonalAverage.time_zonal_average('120km')
6.60±0.05ms 6.57±0.03ms 1 mpas_ocean.ZonalAverage.time_zonal_average('480km')
273M 273M 1 quad_hexagon.QuadHexagon.peakmem_open_dataset
272M 272M 1 quad_hexagon.QuadHexagon.peakmem_open_grid
7.26±0.3ms 6.92±0.1ms 0.95 quad_hexagon.QuadHexagon.time_open_dataset
5.91±0.2ms 5.93±0.08ms 1 quad_hexagon.QuadHexagon.time_open_grid

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

pre-commit.ci autofix

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

There's some spurious asv results from machine variability, but the benchmarks consistenly show peak-mem reductions and some speedups for cross-sections.

@cmdupuis3
cmdupuis3 requested a review from erogluorhan July 20, 2026 23:25
@cmdupuis3
cmdupuis3 marked this pull request as ready for review July 20, 2026 23:26
@cmdupuis3
cmdupuis3 requested a review from rajeeja July 23, 2026 23:26
@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

pre-commit.ci autofix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-benchmark Run ASV benchmark workflow scalability Related to scalability & performance efforts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant