Fix multithreading reproducibility#98
Open
lrobion wants to merge 3 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #19. Multithreaded simulations are now bitwise reproducible.
Multithreaded simulations even using a fixed random seed were not reproducible. This was for three reasons:
Aerosol.cppthat due to floating point non-associativity lead to different results. This floating point noise is magnified because the shear applied to the contrail depends on finding the maximum index ofxODwith depends on multiple non-deterministic reductions. Eventually the numerical noise is sufficient to shift the index of the maxxODwhich changes the shear which then leads to a different simulation grid.This addresses all 3 issues but comes at a signficant performance drop (40% slower simulations on 4h of the ISSL 140 test) due to the non-parallel reductions. This performance drop can be reduced by a lot if we don't repeat these (now slower) reductions multiple times per timestep (see #97).
I think once we address the repeated calculations, we can decide if the performance drop is still significant enough that want to look into this more. We could look into refactoring the reductions to be safe to do in parallel / lowering the bar on reproducibility where results are guaranteed reproducible only if you use the same number of threads. Further work on the performance of this should probably happen after #92 is merged as it also deals with performance in
Aerosol.cpp.