[BUG] Structured Dataset create with remote uri and read directly will fail

[Future-Outlier]

Describe the bug

This case should work.

@task
def return_sd() -> StructuredDataset:
    sd = StructuredDataset(uri="s3://my-s3-bucket/s3_flyte_dir/df.parquet", file_format="parquet")
    print("sd:", sd.open(pd.DataFrame).all())
    return sd

Expected behavior

This should work.

Additional context to reproduce

1. we should upload a parquet file to s3 storage first.
2. FlyteFile maybe has the same bug by @pingsutw

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[JiangJiaWei1103]

Hi @Future-Outlier,

If I'm not mistaken, steps to reproduce the error are described as follows:

1. Upload a simple parquet file to s3 storage:

2. Run the task remotely by executing this command:

pyflyte run --remote reprod.py return_sd

Error Message

Does this match your experience? Thanks.

[Future-Outlier]

Hi @Future-Outlier,

If I'm not mistaken, steps to reproduce the error are described as follows:

1. Upload a simple parquet file to s3 storage:

2. Run the task remotely by executing this command:

pyflyte run --remote reprod.py return_sd

Error Message

Does this match your experience? Thanks.

yes and this will not work in local too.

[Future-Outlier]

Hi @Future-Outlier,

If I'm not mistaken, steps to reproduce the error are described as follows:

1. Upload a simple parquet file to s3 storage:

2. Run the task remotely by executing this command:

pyflyte run --remote reprod.py return_sd

Error Message

Does this match your experience? Thanks.

my suggestion is that we should make local execution work first.

[JiangJiaWei1103]

For the local run, do you mean loading the parquet file from the local file system, instead of s3?

Some Observations

My observation is that self.literal is always set to None and has nothing to do with the input uri.

As suggested here, StructuredDataset should allow data loading by specifying uri. Am I correct?

[Future-Outlier]

For the local run, do you mean loading the parquet file from the local file system, instead of s3?

Some Observations

My observation is that self.literal is always set to None and has nothing to do with the input uri.

As suggested here, StructuredDataset should allow data loading by specifying uri. Am I correct?

I mean make my example above works in the local execution.

1. create a structured dataset from a parquet file from remote storage
2. read it

[JiangJiaWei1103]

Okay, let me try it, thanks!

[JiangJiaWei1103]

Follow-up Issues

After testing some cases, I observe two more errors:

• Open StructuredDataset with df set to a pandas DataFrame

@task
def return_sd() -> StructuredDataset:
    sd = StructuredDataset(
        df=pd.DataFrame({
            "name": ["hanru", "jiawei"],
            "height": [190, 172]
        })
    )
    print("sd:", sd.open(pd.DataFrame).all())
    return sd

• Open StructuredDataset with local uri

@task
def return_sd() -> StructuredDataset:
    sd = StructuredDataset(
        uri="./tmp/df.parquet",
        file_format="parquet"
    )
    print("sd:", sd.open(pd.DataFrame).all())
    return sd

Both of the cases are expected to work. I think they're highly related to this issue. But, I will fix the main error in the original proposal first.

[Future-Outlier]

Very nice, let's push it this week, will reach out you

[wild-endeavor]

in the follow-up issues, i think the second one is more important than the first. The first is a bit odd, esp if the type of the dataframe is different (polars, etc.)

Yeah, to what @pingsutw said, this issue affects files and folders also. cc @eapolinario since he's trying to spend some time on this. I think this is not so much a bug (it is that too) as it is a failure in ux design.

[JiangJiaWei1103]

Hi @wild-endeavor,

Thanks for your reply. The newly merged PR focuses on solving the local run of StructuredDataset with uri only. I'm open to any further progress on related issues in the future!