Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix LocalFileSystem with range request that ends beyond end of file #6751

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

kylebarron
Copy link
Contributor

Which issue does this PR close?

Closes #6749.

Rationale for this change

Make LocalFileSystem act the same as other object store backends.

What changes are included in this PR?

Allow LocalFileSystem to return a satisfiable byte range beyond the end of the file.

Add new test for satisfiable and non-satisfiable byte range.

Are there any user-facing changes?

No.

@github-actions github-actions bot added the object-store Object Store Interface label Nov 18, 2024
Copy link
Contributor

@etseidl etseidl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like several tests are failing because neither the seek nor the read return an error when seeking/reading beyond the end of the file. The Seek and Read traits seem to allow operations beyond EOF...read in particular will simply return 0 rather than produce an error. Perhaps read_range should not test the number of bytes returned at all and return what it can, leaving it up to the caller to detect when a read returns fewer bytes than requested and act accordingly. 🤷

Edit:
Ok, I read the discord discussion, so ignore the above. Instead, how about getting the actual file length from the File metadata, and then test that range.start is not beyond that?

    let file_len = metadata.len();
    ensure!(
        file_len > range.start as u64,
        OutOfRangeSnafu {
            path,
            expected: range.start,
            actual: file_len as usize
        }
    );

    let to_read = range.end - range.start;
    let seek_idx = file
        .seek(SeekFrom::Start(range.start as u64))
        .context(SeekSnafu { path })?;

ensure!(
read == to_read,
seek_idx == range.start as u64,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears this will always be true, even if seeking beyond EOF.

Comment on lines +914 to +916
file.take(to_read as u64)
.read_to_end(&mut buf)
.context(UnableToReadBytesSnafu { path })?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, this read will simply return 0 when reading beyond EOF.

@alamb
Copy link
Contributor

alamb commented Nov 22, 2024

🤔 the CI failures appear to be real

@tustvold tustvold marked this pull request as draft December 12, 2024 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LocalFileSystem errors with satisfiable range request
3 participants