How can I read a pickle (.pkl) file inside the flyte task?

[dnyaneshwar-patil]

I have some training data stored in pickle files and I would like to read those files in flyte task.

def read_pickle_files():
     import pickle
     with open('path/tp/file.pkl', 'rb') as f:
         data = pickle.load(f)

This task does not give out any logs or exceptions. however, it simply fails.

Is there any way to make this work?

[dosubot[bot]]

Hey @dnyaneshwar-patil! Great to see you diving into the intricacies of Flyte again. Hope you're doing fantastic!

To read a pickle file within a Flyte task, you need to ensure that the objects are being serialized and deserialized between the same Python versions, as pickle files are only compatible with the same version of Python they were created with.

You can also consider using the FlytePickle transformer for any unrecognized type hints to enable the serialization/deserialization of Python values to/from a pickle file. This approach allows you to work with custom objects without needing to register a custom transformer. However, for optimal performance, it's recommended to use Python types supported by Flyte or register a custom transformer, as using pickle types can result in lower performance.

Here's an example demonstrating how to utilize custom objects (in this case, a Superhero class) with Flyte tasks and workflows, which implicitly uses the FlytePickle transformer for serialization/deserialization:

from flytekit import task, workflow

class Superhero:
    def __init__(self, name, power):
        self.name = name
        self.power = power

@task
def welcome_superhero(name: str, power: str) -> Superhero:
    return Superhero(name, power)

@task
def greet_superhero(superhero: Superhero) -> str:
    return f"👋 Hello {superhero.name}! Your superpower is {superhero.power}."

@workflow
def superhero_wf(name: str = "Thor", power: str = "Flight") -> str:
    superhero = welcome_superhero(name=name, power=power)
    return greet_superhero(superhero=superhero)

This code snippet demonstrates how to define tasks and workflows with custom types in Flyte, leveraging the FlytePickle transformer for handling the serialization and deserialization process. You can find more details in the Flyte documentation.

If your task is still failing, it would be helpful to have more information. Could you provide the specific error messages or logs that you are receiving when the task fails? Also, is the pickle file accessible from the location specified in the task? This information will help us better understand the issue and provide a more accurate solution.

Sources

• docs/user_guide/data_types_and_io/pickle_type.md
• flytepropeller/pkg/compiler/test/testdata/snacks-core/168_core.type_system.flyte_pickle.greet_1_task.yaml
• flyteadmin/pkg/manager/impl/task_execution_manager.go

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dnyaneshwar-patil]

Thank you for the response.

I checked and pickle file that i am reading has same python version as that of the container I am using.
The pickle file is also accessible to the flyte. I have checked permissions and task has all the read permissions.

As I mentioned earlier, I do not get any logs from the task or any exception when I try to read it.

Any other file format (like JSON) works.

[kumare3]

do you have a reproducible example?