MHD reproducibility #648
Open
MHD reproducibility #648
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit contains a new EMF correction method, taking a maximum value among coinciding edges betweem MeshBlocks. Previously we simply used average, but such operations are not reproducible because they depend on the ordering of arrival of MPI communications.
|
I will add a note on Wiki as the code becomes different from what is written on the paper once this is merged. |
|
|
I'll look into it. |
|
As this change may affect many users' calculations, I am not going to merge this soon. I'd appreciate if many users could test it. |
felker
reviewed
Feb 20, 2025
| @@ -298,7 +278,12 @@ void FaceCenteredBoundaryVariable::SetEMFShearingBoxBoundaryCorrection() { | |||
| void FaceCenteredBoundaryVariable::ClearEMFShearing(EdgeField &work) { | |||
| AthenaArray<Real> &e2 = work.x2e; | |||
| AthenaArray<Real> &e3 = work.x3e; | |||
| e2.ZeroClear(); | |||
| e3.ZeroClear(); | |||
| constexpr Real m = (sizeof(Real) == sizeof(float)) ? -FLT_MAX : -DBL_MAX; | |||
There was a problem hiding this comment.
the C++ way of doing this is much better:
constexpr Real m = -std::numeric_limits<Real>::max();I noticed that the C-style approach somehow made it through to orbital advection and FC flux correction files in the past:
❯ grep -nri "flt_max" ./
./orbital_advection/default_orbital_velocity.cpp:11:#include <cfloat> // FLT_MAX, FLT_MIN
./orbital_advection/orbital_remapping.cpp:11:#include <cfloat> // FLT_MAX
./orbital_advection/set_orbital_advection.cpp:11:#include <cfloat> // FLT_MAX
./orbital_advection/calculate_orbital_advection.cpp:11:#include <cfloat> // FLT_MAX
./orbital_advection/orbital_advection.cpp:11:#include <cfloat> // FLT_MAX, FLT_MIN
./orbital_advection/orbital_advection.cpp:435: vK_max = -(FLT_MAX);
./orbital_advection/orbital_advection.cpp:436: vK_min = (FLT_MAX);
./orbital_advection/orbital_advection.cpp:512: min_dt = (FLT_MAX);
./bvals/fc/flux_correction_fc.cpp:1224: constexpr Real m = (sizeof(Real) == sizeof(float)) ? -FLT_MAX : -DBL_MAX;
would you mind changing them?
There was a problem hiding this comment.
Please change them. Regarding OA, I just opened another PR about it, so probably it is better to fix them there.
felker
reviewed
Feb 20, 2025
| @@ -9,6 +9,7 @@ | |||
| //======================================================================================== | |||
|
|
|||
| // C headers | |||
| #include <float.h> | |||
There was a problem hiding this comment.
Suggested change
| #include <float.h> | |
| #include <float.h> |
see below, use #include <limits> (or at a minimum, cfloat)
felker
requested changes
Feb 20, 2025
|
I agree that the C++ style is much better. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit improves reproducibility of parallel MHD simulations with a new EMF correction method, taking a maximum value among coinciding edges between MeshBlocks. See issue #645.
Prerequisite checklist
Description
Previously we simply used average, but such operations are not reproducible because they depend on the ordering of arrival of MPI communications. This implementation is simpler than the previous one, as it does not need to count the number of coinciding edges.
Note 1: this improves the reproducibility between runs, but it does NOT reproduce calculations with different MeshBlock configuration (e.g. 1 MeshBlock vs multiple MeshBlocks).
Note 2: the operation can be minimum instead of maximum, or max/min in absolute values. I even tried switching max/min at every cycle, but I could not find any meaningful difference between them. All we need is to use exactly same EMF for coinciding edges.
Note 3: I now feel this implementation is better than the previous one, but I'm not 100% sure yet.
This also contains minor leftover fixes.