Preventing memory overflow

[MarcT0K]

In my current experiments, I am trying to scale the parameters until the memory overflows. My goal is more specifically to compare two algorithms and identify the point at which they cannot be executed any more.

The tricky part is then to control the memory overflowing to avoid the computer to be blocked. The OS might kill the script at some point but it is better to control this process ourselves, especially if we want to execute other tasks (e.g., writing the results in a csv) after the overflowing).

The resource built-in library is helpful to set a limit on the memory usage. If the memory exceeds a certain limit, a MemoryError will be raised. All this looks great but it gets more complicated since MPyC operations are asynchronous.

To simulate an overflow, one can use the following code:

import resource

from mpyc.runtime import mpc


async def overflow(n=10**9):
    sectype = mpc.SecInt()
    x = [mpc.input(sectype(i), senders=0) for i in range(n)]
    await mpc.gather(x)


async def main():
    _prev_soft, prev_hard = resource.getrlimit(resource.RLIMIT_AS)

    with open("/proc/meminfo", "r") as mem:  # We estimate the free memory
        free_memory = 0
        for i in mem:
            sline = i.split()
            if str(sline[0]) in ("MemFree:", "Buffers:", "Cached:"):
                free_memory += int(sline[1])

    resource.setrlimit(
        # We limit the memory usage to prevent memory overflow from crashing the PC
        resource.RLIMIT_AS,
        (int(free_memory * 1024 * 0.3 / len(mpc.parties)), prev_hard),
    )

    await mpc.start()
    try:
        await overflow(10**6)
    except MemoryError:
        resource.setrlimit(
            # We increase the limit to avoid MemoryError while handling MemoryError
            resource.RLIMIT_AS,
            (int(free_memory * 1024 * 0.9 / len(mpc.parties)), prev_hard),
        )
        print("Caught Memory error")

    await mpc.shutdown()


if __name__ == "__main__":
    mpc.run(main())

All the code is executed normally until the shutdown. Indeed, we can catch the MemoryError but it seems complicated to pursue the MPC protocol afterwards. I've started investigating and it seems it comes from a desynchronization between the parties: some parties may have been stopped by the MemoryError while others never crashed. Hence, there may be one party waiting for messages that will never arrive. More generally, the MemoryError can occur at anytime during the coroutine execution. Hence, it is hard to recover from a state on which we have little control.

From these first investigations, I feel like I have two options:

1. Make invasive changes into MPyC to add a layer of synchronization in the exception handling (i.e., if a party crashes for whatever reason, all of them should have the same exception).
2. Run all my experiments in a separate subprocess that I let it crash due to the MemoryError. The calling process would them handle CSV changes in case of crash.

Feel free to contribute to this discussion with potential ideas you may have to solve this problem. For now, I'll work with the second option which requires much less work than the first option. If someone else is interested by such a feature, I may isolate my code and improve it to make it easy to use (e.g., using pickle to simplify the subprocess creation).