Hackathon 2025/03/04

[github-actions[bot]]

Reporting out on earthaccess hack days. Use the 'comment' button at the very bottom to send a message. Additionally, consider sending issues and PRs relevant to your work to help make the job of future readers easier. It is okay to duplicate information here! Use the reply feature to have a discussion under any comment. Enjoy!

[ateucher]

Andy played with using earthaccess from R with newest Reticulate - it does all of the python setup and dependency management on the fly, in the background using uv. Followed the "Ho To" from the cookbook accessing data with Python

This worked totally seamlessly:

# pak::pak("reticulate")

library(reticulate)

py_require("earthaccess")
py_require("xarray")

# xarray optional dependencies
py_require("h5netcdf")
py_require("netCDF4")
py_require("zarr")
py_require("dask")
py_require("matplotlib")

earthaccess <- import("earthaccess")
xr <- import("xarray", "xr")

matplotlib <- import("matplotlib", convert = TRUE)
matplotlib$use("Agg", force = TRUE)

plt <- import("matplotlib.pyplot", as = "plt")

auth <- earthaccess$login(strategy = "netrc")

#Access land ice height from ATLAS/ICESat-2 V005 (10.5067/ATLAS/ATL06.005), searching for data over western Greenland coast over two weeks in July 2022. The data are provided as HDF5 granules (files) that span about 1/14th of an orbit.

results <- earthaccess$search_data(
  short_name="ATL06",
  version="006",
  cloud_hosted=TRUE,
  # declare `temporal` and `bounding_box` as tuples
  temporal = tuple("2022-07-17", "2022-07-31"),
  bounding_box = tuple(-51.96423, 68.10554, -48.71969, 70.70529)
)

ds <- xr$open_mfdataset(
  earthaccess$open(results),
  group = '/gt1l/land_ice_segments'
)

ds
#> <xarray.Dataset> Size: 11MB
#> Dimensions:                (delta_time: 249580)
#> Coordinates:
#>   * delta_time             (delta_time) datetime64[ns] 2MB 2022-07-18T01:00:4...
#>     latitude               (delta_time) float64 2MB dask.array<chunksize=(10000,), meta=np.ndarray>
#>     longitude              (delta_time) float64 2MB dask.array<chunksize=(10000,), meta=np.ndarray>
#> Data variables:
#>     atl06_quality_summary  (delta_time) int8 250kB dask.array<chunksize=(10000,), meta=np.ndarray>
#>     h_li                   (delta_time) float32 998kB dask.array<chunksize=(10000,), meta=np.ndarray>
#>     h_li_sigma             (delta_time) float32 998kB dask.array<chunksize=(10000,), meta=np.ndarray>
#>     segment_id             (delta_time) float64 2MB dask.array<chunksize=(10000,), meta=np.ndarray>
#>     sigma_geo_h            (delta_time) float32 998kB dask.array<chunksize=(10000,), meta=np.ndarray>
#> Attributes:
#>     Description:  The land_ice_height group contains the primary set of deriv...
#>     data_rate:    Data within this group are sparse.  Data values are provide...

^{Created on 2025-03-05 with reprex v2.1.1}

Note - it was quite slow; xr$open_mfdataset() took several minutes. Is that normal @betolink @mfisher87 @asteiker?

Update, from @betolink in Slack:

it's slow because the backend Xarray uses for netcdf is opening each file to get the metadata it needs, in Opendap and Zarr that metadata was consolidated upfront

[betolink]

Really interested in comparing the performance of R packages vs this pyrequire approach, in the virtualized zarr form. With MUR (ATL06 won't open easily with xarray multi file dataset because is an L2 swath dataset) Thanks for looking into this @ateucher !