Parallel Process Distribution of Refined Levels

[aerojpapp]

Hello,

Is there any way to control to which process refined levels are distributed? More specifically, is there anyway to force only 1 core per box if the number of refined boxes is less than the available cores? I've noticed that AMReX will often double up boxes on a core. For instance, if I've reserved 16 cores and I have say 14 refined boxes, some processors will get 2 boxes while others will get only 1 or 0.

Thanks,

John

[asalmgren]

Re the number of boxes -- with the default setting of refine_grid_layout = true , if there are fewer boxes than MPI processes, the code will subdivide some of the boxes until Ngrids >= Ncores This isn't done one-by-one to get to Ngrids == Ncores, so yes, you can definitely end up with more than one box per rank. The load balancing procedure will do the best job possible of distributing the work. It should never be the case that you have ranks with no boxes, unless you set refine_grid_layout to false If you do have such a case, could you let us know how we can replicate it to see what's going on?

…

On Mon, Nov 8, 2021 at 10:56 AM aerojpapp ***@***.***> wrote: Hello, Is there any way to control to which process refined levels are distributed? More specifically, is there anyway to force only 1 core per box if the number of refined boxes is less than the available cores? I've noticed that AMReX will often double up boxes on a core. For instance, if I've reserved 16 cores and I have say 14 refined boxes, some processors will get 2 boxes while others will get only 1 or 0. Thanks, John — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2465>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YQV4VS2IV7R363TZ3LULAMNPANCNFSM5HTNGM2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Ann Almgren Senior Scientist; CCSE Group Lead Pronouns: she/her/hers

[aerojpapp]

Thanks for the quick answer. After looking more closely, AMReX is doing what it is supposed to be doing. I wasn't identifying the refined blocks properly and assuming there where no blocks on some processors when, in fact, there were.

I'm not currently setting the 'refine_grid_layout' value so I assume it is true by default. If I unset it, will I always get NGrids <= NCores even if that means no boxes on some processors?

Thanks,

John

[asalmgren]

if you set refine_grid_layout to false then you will get however many grids the gridding routine makes (depends on max_grid_size, blocking_factor, etc) -- so it may be that Ngrids < Ncores or Ngrids >= Ncores -- see https://amrex-codes.github.io/amrex/docs_html/GridCreation.html#sec-grid-creation for more details of exactly how the grids are constructed

…

On Wed, Nov 10, 2021 at 8:59 AM aerojpapp ***@***.***> wrote: Thanks for the quick answer. After looking more closely, AMReX is doing what it is supposed to be doing. I wasn't identifying the refined blocks properly and assuming there where no blocks on some processors when, in fact, there were. I'm not currently setting the 'refine_grid_layout' value so I assume it is true by default. If I unset it, will I always get NGrids <= NCores even if that means no boxes on some processors? Thanks, John — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2465 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YQPMKQEUUKESVQOQX3ULKQHFANCNFSM5HTNGM2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Ann Almgren Senior Scientist; CCSE Group Lead Pronouns: she/her/hers

[aerojpapp]

Thanks again. Would a feature to prevent NGrids > NCores be possible to implement or can it be implemented through specifying max_grid_size and blocking_factor (or some other parameter)? My problem requires all grids to be solved simultaneously and if is more than one block on a processor, I have to use threading, which I think may be less efficient then just using bigger grid blocks to keep a 1:1 ratio of grids to cores.

John

[asalmgren]

This is something that would be easy to do on a small scale but hard to do optimally on a large-scale, I suspect. If I understand correctly -- you want to have exactly Ngrids == Ncores on each level and are willing to compromise on the other criteria (like blocking factor)?

…

On Wed, Nov 10, 2021 at 9:18 AM aerojpapp ***@***.***> wrote: Thanks again. Would a feature to prevent NGrids > NCores be possible to implement or can it be implemented through specifying max_grid_size and blocking_factor (or some other parameter)? My problem requires all grids to be solved simultaneously and if is more than one block on a processor, I have to use threading, which I think may be less efficient then just using bigger grid blocks to keep a 1:1 ratio of grids to cores. John — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2465 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YSSZZSZI6JDHYRZTLLULKSOXANCNFSM5HTNGM2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Ann Almgren Senior Scientist; CCSE Group Lead Pronouns: she/her/hers

[aerojpapp]

Not quite. I just can't have NGrids > NCores. I can have NGrids<=NCores. I already have code in place to account for cores that have no grid blocks so that isn't a problem.

John

[WeiqunZhang]

My problem requires all grids to be solved simultaneously and if is more than one block on a processor

Could you explain why you cannot have more than one block on a process?

[aerojpapp]

It might be complicated to explain. I've buried AMReX inside a NS solver that uses an ADI implicit scheme. I need to synchronize ghost cell data between directional sweeps, which means that I either have to place the AMReX box loop inside the sweep for processors that have more than one refined box or run in parallel threaded mode if the ADI sweep loop is inside the AMReX box loop. The former method would require extensive reorganization of the underlying solver so I've used the latter and have seen a hit in computational efficiency (about 10x). I'm trying to discern if the cause is the overloaded threading or if it just is that slow because to the ghost cell communication. If I can limit NGrid<=NCores, I can test both.

John

[asalmgren]

Thinking out loud -- the way the grid creation works, we start with a single bounding box -- the smallest box that contains all the cells tagged for refinement -- then we cut that box...then cut the resultant boxes .. and so on until the criteria are met. So if you're willing to sacrifice other criteria, we could just add an option to stop cutting if the cut would result in > Ncores boxes. That should work in principle and for most cases ... I haven't thought about it long enough to know if there are some gotchas that might come up.

…

On Wed, Nov 10, 2021 at 11:00 AM aerojpapp ***@***.***> wrote: Not quite. I just can't have NGrids > NCores. I can have NGrids<=NCores. I already have code in place to account for cores that have no grid blocks so that isn't a problem. John — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2465 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YTIT5YAETYNYBTRTSDULK6OJANCNFSM5HTNGM2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Ann Almgren Senior Scientist; CCSE Group Lead Pronouns: she/her/hers

[aerojpapp]

Where is that done? I can take a look at the source and see if I can modify.

Thanks,

John

[asalmgren]

In amrex/Src/AmrCore/AMReX_AmrMesh.cpp we have AmrMesh::MakeNewGrids which does the whole process of creating tagged cells and then creating the new BoxArray. If you go to about line 680 in that file, you'll see the call to "chop" which repeatedly "chops" up the boxarray -- starting with the initial bounding box -- until the criteria are met. chop itself lives in Src/AmrCore/AMReX_Cluster.cpp -- see line 530 that is where the test on grid efficiency is -- that is one thing you may not be able to satisfy with Ngrids <= Ncores You should be able to follow the logic from there -- I didn't get far enough to figure out exactly the best place to intersect the logic to ensure Ngrids <= Ncores but it should be fairly straightforward to figure out

…

On Fri, Nov 12, 2021 at 10:10 AM aerojpapp ***@***.***> wrote: Where is that done? I can take a look at the source and see if I can modify. Thanks, John — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2465 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YVR5MQG5QCISTT6JN3ULVKCXANCNFSM5HTNGM2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Ann Almgren Senior Scientist; CCSE Group Lead Pronouns: she/her/hers

[aerojpapp]

Thanks, I'll check it out and let you know.

John