Skip to content

Commit

Permalink
Add comment about offloaded literals files and how they are uploaded …
Browse files Browse the repository at this point in the history
…to the metadata bucket

Signed-off-by: Eduardo Apolinario <[email protected]>
  • Loading branch information
eapolinario committed Nov 8, 2024
1 parent 37b2bb4 commit e25496d
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion flytekit/bin/entrypoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,10 @@ def _dispatch_execute(
if lit.ByteSize() >= min_offloaded_size:
logger.debug(f"Literal {k} is too large to be inlined, offloading to metadata bucket")

# This file will hold the offloaded literal and will be written to the output prefix
# alongside the regular outputs.pb, deck.pb, etc.
# N.B.: by construction `offloaded_filename` is guaranteed to be unique
offloaded_filename = f"{k}_offloaded_metadata.pb"

offloaded_literal = _literal_models.Literal(
offloaded_metadata=_literal_models.LiteralOffloadedMetadata(
uri=f"{output_prefix}/{offloaded_filename}",
Expand Down

0 comments on commit e25496d

Please sign in to comment.