Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve memory of parallel fitGLS_partition #15

Open
morrowcj opened this issue Dec 16, 2022 · 1 comment
Open

improve memory of parallel fitGLS_partition #15

morrowcj opened this issue Dec 16, 2022 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@morrowcj
Copy link
Owner

morrowcj commented Dec 16, 2022

Problem

The parallel partitioned GLS is driven by the function MC_GLSpart(). This function utilizes foreach(i = 1:npart, ...) %dopar% {...} syntax. This formulation has the entire dataset imported on each instance (thread). That leads to memory usage snowballing quite quickly (ncores $\times$ the size of the data object).

Solution

foreach() accepts an iterator that allows data to be constructed on the fly. In short, this could allow only the data from the partition of interest to be imported for a given instance. The upshot is that the total memory usage shouldn't be much greater than the total size of the original object. So, we should swap i = 1:npart with an iterator to provide partitions.

@morrowcj morrowcj added the enhancement New feature or request label Dec 16, 2022
@morrowcj morrowcj self-assigned this Dec 16, 2022
@morrowcj
Copy link
Owner Author

The recommended solution to #13 may also be useful here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant