Skip to content

Refactor windowed arrays to use a numpy backed DataArray cache - #2846

Merged
wyatt-fluidnumerics merged 6 commits into
mainfrom
save-windowed-array-cache-as-dataarray
Sep 3, 2026
Merged

Refactor windowed arrays to use a numpy backed DataArray cache#2846
wyatt-fluidnumerics merged 6 commits into
mainfrom
save-windowed-array-cache-as-dataarray

Conversation

@wyatt-fluidnumerics

@wyatt-fluidnumerics wyatt-fluidnumerics commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Description

This improvement is largely outlined in #2843. The goal is to remove the current codes dependence on a np.stack call in WindowedArray.isel() which causes a large in memory copy. This implementation replaces the existing dictionary cache with a numpy backed DataArray, which allows for direct indexing and removes the need to reconstruct a DataArray from the dictionary cache on every call to .isel(). This results in significant performance improvement for large datasets (as much 20x or more depending on size and advection scheme).

Additionally, these changes mean that in theory WindowedArray now has all the needed functionality to support non-synchronous clocks. In practice however, this is likely not a very practical use case as for large datasets holding even a few time levels in the cache can lead to OOM errors on many machines.

Checklist

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.
    • Describe how you used it (e.g., by pasting your prompt): Claude code was used to write code to benchmark this change against the previous version and create the plot in Windowed Array Improvements/Optimization #2843. I thoroughly reviewed the benchmarking code. It was also used to write parts of the new implementation, though significant changes and rewrites to those components were made by myself.

@erikvansebille erikvansebille left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@wyatt-fluidnumerics
wyatt-fluidnumerics merged commit 9dc6bb4 into main Sep 3, 2026
24 checks passed
@wyatt-fluidnumerics
wyatt-fluidnumerics deleted the save-windowed-array-cache-as-dataarray branch September 3, 2026 16:53
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Parcels development Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Windowed Array Improvements/Optimization

2 participants