Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to identify out of memory errors and print corresponding error message #58

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Try to identify out of memory errors and print corresponding error message #58

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5b87029
Log error return code
pagmatt May 25, 2022
15febef
Identify SIGKILL and log ad hoc error msg
pagmatt May 25, 2022
838d5af
Switch to self-def ret code constant
pagmatt May 25, 2022
93867cf
Fix ident
pagmatt May 25, 2022
3d03c1f
Improve err msg
pagmatt May 25, 2022
8e863e2
Further improve err msg
pagmatt May 25, 2022
09452d1
Self-review
pagmatt May 25, 2022
0b36785
Most DM comments
pagmatt May 30, 2022
805cf72
Proper string formatting
pagmatt May 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 26 additions & 15 deletions sem/runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
import sem.utils

from tqdm import tqdm
from typing import Final

SIGKILL_CODE: Final = -9 # POSIX return code which usually corresponds to out of memory events.
Comment on lines +10 to +12
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary to import typing.Final here? We don't use typing anywhere else in the project.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not necessary, it just seemed "good practice" to use it for marking the variable (almost) as a constant. At least, that was my understanding from the PEP guideline which I linked above. If you prefer to limit the imports though I can remove it!


class SimulationRunner(object):
"""
Expand Down Expand Up @@ -327,21 +329,30 @@ def run_simulations(self, parameter_list, data_folder, stop_on_errors=False):
if return_code != 0:
with open(stdout_file_path, 'r') as stdout_file, open(
stderr_file_path, 'r') as stderr_file:
complete_command = sem.utils.get_command_from_result(self.script, current_result)
complete_command_debug = sem.utils.get_command_from_result(self.script, current_result, debug=True)
error_message = ('\nSimulation exited with an error.\n'
'Params: %s\n'
'Stderr: %s\n'
'Stdout: %s\n'
'Use this command to reproduce:\n'
'%s\n'
'Debug with gdb:\n'
'%s'
% (parameter,
stderr_file.read(),
stdout_file.read(),
complete_command,
complete_command_debug))
common_error_message = ('Params: %s\n'
'Stderr: %s\n'
'Stdout: %s\n'
'Return code: %d\n'
% (parameter,
stderr_file.read(),
stdout_file.read(),
return_code))
if return_code == SIGKILL_CODE:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be worth it to always print return_code when it's not 0, even when it's not exactly -9. Can you add this information to the common_error_message?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

error_message = '\nSimulation was killed. Possible causes may include an out of memory error.\n' + \
'Check kernel logs (dmesg, for instance) to confirm.\n' + \
common_error_message
else:
complete_command = sem.utils.get_command_from_result(self.script, current_result)
complete_command_debug = sem.utils.get_command_from_result(self.script, current_result, debug=True)
error_message = '\nSimulation exited with an error.\n' + \
common_error_message + \
('Use this command to reproduce:\n'
'%s\n'
'Debug with gdb:\n'
'%s'
% (complete_command,
complete_command_debug))

if stop_on_errors:
raise Exception(error_message)
print(error_message)
Expand Down