You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The parallel partitioned GLS is driven by the function MC_GLSpart(). This function utilizes foreach(i = 1:npart, ...) %dopar% {...} syntax. This formulation has the entire dataset imported on each instance (thread). That leads to memory usage snowballing quite quickly (ncores $\times$ the size of the data object).
Solution
foreach() accepts an iterator that allows data to be constructed on the fly. In short, this could allow only the data from the partition of interest to be imported for a given instance. The upshot is that the total memory usage shouldn't be much greater than the total size of the original object. So, we should swap i = 1:npart with an iterator to provide partitions.
The text was updated successfully, but these errors were encountered:
Problem
The parallel partitioned GLS is driven by the function$\times$ the size of the data object).
MC_GLSpart()
. This function utilizesforeach(i = 1:npart, ...) %dopar% {...}
syntax. This formulation has the entire dataset imported on each instance (thread). That leads to memory usage snowballing quite quickly (ncoresSolution
foreach()
accepts aniterator
that allows data to be constructed on the fly. In short, this could allow only the data from the partition of interest to be imported for a given instance. The upshot is that the total memory usage shouldn't be much greater than the total size of the original object. So, we should swapi = 1:npart
with an iterator to provide partitions.The text was updated successfully, but these errors were encountered: