Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hil: add option for reusing existing rts #269

Closed
wants to merge 2 commits into from

Conversation

AlexKaravaev
Copy link
Contributor

s3 urls are unique for given commit, we might want to reuse them for gh workflows, but currently since we manipulate filename inside orb-hil we cannot supply correct rts-path (We can in theory but that's ugly)

Copy link
Collaborator

@TheButlah TheButlah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not add functionality that is brittle. Unless we perform a checksum validation on the file, and retrieve the canonical sha from s3, I don't think we should reuse any files.

Do you think you could look into implementing sha retrieval from s3?

@AlexKaravaev
Copy link
Contributor Author

yep, will do so. It should be fairly easy.

@AlexKaravaev
Copy link
Contributor Author

It turned out to be trickier than I expected:

  • AWS requires the user to set a checksum when uploading; otherwise, it splits the file into multiple parts, and "Amazon S3 concatenates the bytes for the MD5 digests together and then calculates the MD5 digest of these concatenated values." (See AWS Documentation).

  • The number of parts seems constant in our case but essentially depends on the version of the AWS S3 CLI. For me, the chunk size was 8192 KB.

  • I don’t want to hard-code these numbers, but I haven’t found a way to retrieve them reliably at runtime. I suggest we close this for now and revisit it later. The optimization here is that we avoid downloading the RTS for the same commit, but in practice, it probably won’t happen often that we need to download RTS in CI for the same commit. Plus, we only save about 6 minutes with this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants