[BUG] Raise an error if fast-package.tar.gz is too big

[pingsutw]

Describe the bug

request failed to upload the tar file if it's too big.

Expected behavior

We should check the file size before uploading and raise a nicer error.

Additional context to reproduce

import random
import string

from flytekit import task, workflow


@task(enable_deck=True)
def t1(a: int) -> int:
    return a + 1


@workflow
def wf() -> int:
    return t1(a=3)


if __name__ == '__main__':
    def create_large_file(filename, size_in_mb):
        # Size of the file in megabytes
        size_in_bytes = size_in_mb * 1024 * 1024
        chunk = bytes(''.join(random.choices(string.ascii_uppercase + string.digits, k=1024 * 1024)), 'utf-8')

        with open(filename, 'wb') as f:
            for _ in range(size_in_bytes // len(chunk)):
                f.write(chunk)

            # Write the remaining bytes if any
            remaining_bytes = size_in_bytes % len(chunk)
            if remaining_bytes:
                f.write(b'0' * remaining_bytes)

    create_large_file('/Users/kevin/git/flytekit/flyte-example/test/dummy', 1024 * 5)

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

The issue you described is related to the failure of uploading a tar file if it exceeds a certain size limit. This can be addressed by checking the file size before attempting the upload and raising a more user-friendly error if the file is too large.

To implement this, you can add a file size check in the flytekit codebase before the upload request. If the file size exceeds the allowed limit, raise an appropriate error message.

Would you like more details on how to implement this in the flytekit codebase?

Sources:

• [House keeping] remove setting max size bytes in node context  #5092
• [BUG] [LIMIT_EXCEEDED] limit exceeded. 5.357096mb > 2mb. with Flyte-binary Chart #4951

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[pingsutw]

fixed by flyteorg/flytekit#2641