Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORM: Add get_size_on_disk method to RemoteData #6584

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

GeigerJ2
Copy link
Contributor

@GeigerJ2 GeigerJ2 commented Oct 15, 2024

Required for PR #6578.

By default, the get_size_on_disk method calls the method get_size_on_disk_du that uses du to obtain the total directory size in bytes. The output will eventually be formatted in a human-readable way. If the call to du fails for whatever reason, recursive lstat is being used, though, that is discouraged due to (copied from _get_size_on_disk_lstat docstring):

"Note that even if a file is only 1 byte, on disk, it still occupies one full disk block size. As such, getting accurate measures of the total expected size on disk when retrieving a RemoteData is not straightforward with lstat, as one would need to consider the occupied block sizes for each file, as well as repository metadata. Thus, this function only serves as a fallback in the absence of the du command."

I further extended the existing tests for RemoteData to use both, LocalTransport, as well as SshTransport, and test for the functionality added in this PR.

Pinging also @npaulish, as she's currently working on retrieving RemoteData objects.

Copy link

codecov bot commented Oct 15, 2024

Codecov Report

Attention: Patch coverage is 79.77528% with 18 lines in your changes missing coverage. Please review.

Project coverage is 77.93%. Comparing base (baf8d7c) to head (a0047ed).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/aiida/orm/nodes/data/remote/base.py 84.62% 10 Missing ⚠️
src/aiida/cmdline/commands/cmd_data/cmd_remote.py 50.00% 7 Missing ⚠️
src/aiida/common/utils.py 90.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6584      +/-   ##
==========================================
+ Coverage   77.92%   77.93%   +0.02%     
==========================================
  Files         563      563              
  Lines       41668    41755      +87     
==========================================
+ Hits        32464    32537      +73     
- Misses       9204     9218      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@GeigerJ2 GeigerJ2 changed the title Add get_total_size_on_disk method to RemoteData. Add get_total_size_on_disk method to RemoteData Oct 17, 2024
@giovannipizzi
Copy link
Member

This is maybe machine-dependent, but rather than going via our API (that is more robust, but definitely going to be slower, I think) have a first "fast" option just running du -s and parsing the output (but careful about units! E.g. it uses "blocks", on some systems it's 512 or 2048bytes!! And if it fails, fall back to your solution?

@GeigerJ2
Copy link
Contributor Author

GeigerJ2 commented Oct 17, 2024

Note to self to run du via exec_command_wait method from transport.

@GeigerJ2 GeigerJ2 requested a review from khsrali November 21, 2024 14:53
@GeigerJ2 GeigerJ2 force-pushed the feature/remote-data-total-size branch from e801a78 to e0be575 Compare November 21, 2024 14:54
@GeigerJ2 GeigerJ2 marked this pull request as ready for review November 21, 2024 14:54
@GeigerJ2 GeigerJ2 changed the title Add get_total_size_on_disk method to RemoteData Add get_size_on_disk method to RemoteData Nov 21, 2024
@GeigerJ2 GeigerJ2 force-pushed the feature/remote-data-total-size branch from e0be575 to b1a0560 Compare November 21, 2024 15:14
Copy link
Contributor

@khsrali khsrali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @GeigerJ2 please go through more pains that I imposed 😈

src/aiida/common/utils.py Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved

return format_directory_size(size_in_bytes=total_size)

def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this whole function, doesn't really need to be separated I feel.. It's basically only executes a command, and you don't re-use it, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the design to have just one top-level method get_size_on_disk nice, which tries to call the two private methods, in terms of separation of concerns.

Though, it is true that this pollutes the API of RemoteData somewhat... So I'm also fine of either moving the private methods to some utils module, or merging them. Maybe @unkcpz can comment on good coding practices here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which tries to call the two private methods, in terms of separation of concerns.

I'm not sure if I understand, sorry 😬
Since it's not really re-usable by other method, I'd vote for merging it, and avoid over-fictionalizing

src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
@pytest.fixture
def remote_data_ssh(tmp_path, aiida_computer_ssh):
"""Return a non-empty ``RemoteData`` instance."""
# Compared to `aiida_localhost`, `aiida_computer_ssh` doesn't return an actual `Computer`, but just a factory
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙉 aiida_computer_ssh should be compared with aiida_computer_local, which in that sense they are similar.. the issue is we don't have "aiida_ssh" that would return the actual computer instance, lol 🙉

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, exactly. We could add a fixture that actually calls the factory and returns the Computer instance, though I'd rather call it aiida_localhost_ssh, and don't see the need for it right now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yews, so maybe please fix this comment, which I found confusing:

Suggested change
# Compared to `aiida_localhost`, `aiida_computer_ssh` doesn't return an actual `Computer`, but just a factory
# `aiida_computer_ssh` doesn't return an actual `Computer`, but just a factory

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now use the aiida_computer_local fixture instead of aiida_localhost, and actually create the Computer from the factory myself (basically the same single line, which is also the only content of the aiida_localhost fixture), such that the behavior between the two remote_data_[local,ssh] fixtures is exactly the same, and no comment in the code is needed anymore.

Copy link
Contributor Author

@GeigerJ2 GeigerJ2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why my replies via the files GH tab appear as a review, but whatever... ^^

Thanks for the review @khsrali. I implemented most of your proposed changes!

As to our in-person discussion if du or lstat is preferred, I think none of the two is ideal... lstat giving the actual byte-sized content, which is neat and all, but won't correspond to the disk space that will actually be occupied locally by a file, due to the use of blocks for the file system. And du giving the actual occupied disk space on the remote, which, however, might be different from the local file system (due to having a different file system on the local machine, different formatting, different block size, etc.). Hence, the big difference in the file size check in the test. For real-world use cases, with more and larger files, the difference is likely much smaller, and won't matter too much, I think. I'll add a test for a larger file, as well as modify the message that the given size is just an estimate, so that user are aware they should take the value with a grain of salt.

Maybe also @agoscinski with his actual computer science background can weigh in 🫶

src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved

return format_directory_size(size_in_bytes=total_size)

def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the design to have just one top-level method get_size_on_disk nice, which tries to call the two private methods, in terms of separation of concerns.

Though, it is true that this pollutes the API of RemoteData somewhat... So I'm also fine of either moving the private methods to some utils module, or merging them. Maybe @unkcpz can comment on good coding practices here?

src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
@GeigerJ2 GeigerJ2 force-pushed the feature/remote-data-total-size branch from f2f90f9 to 4fcb1ba Compare November 25, 2024 15:41
Copy link
Contributor

@khsrali khsrali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @GeigerJ2! Just a few minor comments..

src/aiida/common/utils.py Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Outdated Show resolved Hide resolved
@pytest.fixture
def remote_data_ssh(tmp_path, aiida_computer_ssh):
"""Return a non-empty ``RemoteData`` instance."""
# Compared to `aiida_localhost`, `aiida_computer_ssh` doesn't return an actual `Computer`, but just a factory
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yews, so maybe please fix this comment, which I found confusing:

Suggested change
# Compared to `aiida_localhost`, `aiida_computer_ssh` doesn't return an actual `Computer`, but just a factory
# `aiida_computer_ssh` doesn't return an actual `Computer`, but just a factory

return format_directory_size(size_in_bytes=total_size)

def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:
"""Connects to the remote folder and returns the total size of all files in the directory recursively in bytes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Connects to the remote folder and returns the total size of all files in the directory recursively in bytes
"""Connects to the remote folder and returns the total size of all files in the directory in bytes

:param transport: Open transport instance
:type transport: Transport
:raises RuntimeError: When `du` command cannot be successfully executed
:return: Total size of directory recursively in bytes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I found the term recursively confusing.. (thought the function does yield )

Suggested change
:return: Total size of directory recursively in bytes.
:return: Total size of directory in bytes (including all it's contents)

:param transport: Open transport instance.
:type transport: Transport
:raises RuntimeError: When `du` command cannot be successfully executed.
:return: Total size of directory recursively in bytes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same:

Suggested change
:return: Total size of directory recursively in bytes.
:return: Total size of directory in bytes (including all it's contents)


return format_directory_size(size_in_bytes=total_size)

def _get_size_on_disk_du(self, full_path: Path, transport: 'Transport') -> int:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which tries to call the two private methods, in terms of separation of concerns.

I'm not sure if I understand, sorry 😬
Since it's not really re-usable by other method, I'd vote for merging it, and avoid over-fictionalizing

@GeigerJ2 GeigerJ2 force-pushed the feature/remote-data-total-size branch from f7dfe7e to ecfa53b Compare December 9, 2024 09:36
Copy link
Contributor

@agoscinski agoscinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes seems like we should split the function into two or provide optional arguments. These two concepts seem to be distinguished by the terms "disk usage" and "apparent size" (see https://stackoverflow.com/a/569485). I think the output of du --apparent-size is the same as with lstat, so you do not need to use lstat

"""
try:
total_size = 0
contents = self.listdir_withattributes(full_path)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this work recursively? what if there are nested subfolders does this still work correctly?

src/aiida/cmdline/commands/cmd_data/cmd_remote.py Outdated Show resolved Hide resolved
src/aiida/orm/nodes/data/remote/base.py Show resolved Hide resolved
"""

try:
retval, stdout, stderr = transport.exec_command_wait(f'du --bytes {full_path}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not work on macos. It seems like macos has no easy way do get the size in bytes (one can get the number of blocks but I don't know how to get the block size). One can get the size in kilobytes du -k. Anyway we do not seem to run CI on macos at the moment so I don't think it is important for this PR

@GeigerJ2
Copy link
Contributor Author

Thanks for the review, @agoscinski! I'm currently still working on this, will ping you once it's again ready for review.

I think the output of du --apparent-size is the same as with lstat, so you do not need to use lstat

The reason I'm providing lstat as a fallback option is if du is not available (e.g., MacOS, as you mentioned, didn't know that ^^), or if exec_command_wait isn't available, which will be the case for FirecREST.

GeigerJ2 and others added 11 commits December 11, 2024 11:59
In addition, extend the tests for `RemoteData` in `test_remote.py`
for the methods added in this PR, as well as parametrize them to run on
a `RemoteData` via local and ssh transport.
In addition, extend the tests for `RemoteData` in `test_remote.py`
for the methods added in this PR, as well as parametrize them to run on
a `RemoteData` via local and ssh transport.
@GeigerJ2 GeigerJ2 force-pushed the feature/remote-data-total-size branch from 58b3248 to 8bcf0f8 Compare December 11, 2024 10:59
@GeigerJ2
Copy link
Contributor Author

OK, this should be ready for a final review, @agoscinski and @khsrali. Also pinging, @mikibonacci, if you want to provide some feedback on the CLI/API for use in AiiDAlab?

@GeigerJ2 GeigerJ2 changed the title Add get_size_on_disk method to RemoteData ORM: Add get_size_on_disk method to RemoteData Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants