Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return StructuredDataset which is a field in a dataclass #3071

Closed
wants to merge 14 commits into from
+77 −0
Closed

Return StructuredDataset which is a field in a dataclass #3071

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
51f6f73
test
arbaobao Jan 20, 2025
675a7fa
remove breakpoint
arbaobao Jan 20, 2025
a3df842
add commit
arbaobao Jan 20, 2025
c7b8f5d
Merge branch 'master' of github.com:flyteorg/flytekit into structured…
arbaobao Feb 3, 2025
2b1ee07
add integration test
arbaobao Feb 4, 2025
1d8f396
fix error
arbaobao Feb 4, 2025
5329247
fix test err
arbaobao Feb 4, 2025
7a4e749
Merge branch 'master' of github.com:flyteorg/flytekit into structured…
arbaobao Feb 7, 2025
3e390e8
test
arbaobao Feb 7, 2025
60142d0
test
arbaobao Feb 7, 2025
d26a6e9
test
arbaobao Feb 7, 2025
123895d
Merge branch 'master' of github.com:flyteorg/flytekit into structured…
arbaobao Feb 14, 2025
c6d9010
increase timeout to see if it failed
arbaobao Feb 14, 2025
13d7e5a
Merge branch 'master' of github.com:flyteorg/flytekit into structured…
arbaobao Feb 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Loading
Loading status checks…
test 
Signed-off-by: Nelson Chen <[email protected]>
@arbaobao
arbaobao committed Feb 7, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
commit 60142d0e644710052e4f9da489c4c85743e4d90a
2 changes: 1 addition & 1 deletion tests/flytekit/integration/remote/test_remote.py
View file
Original file line number Diff line number Diff line change
@@ -893,7 +893,7 @@ def test_attr_access_dc_sd():
file_transfer = SimpleFileTransfer()
remote_file_path = file_transfer.upload_file(file_type="parquet")

execution_id = run("attr_access_dc_sd.py", "wf")
execution_id = run("attr_access_dc_sd.py", "wf", "--uri", remote_file_path)
remote = FlyteRemote(Config.auto(config_file=CONFIG), PROJECT, DOMAIN)
execution = remote.fetch_execution(name=execution_id)
execution = remote.wait(execution=execution, timeout=datetime.timedelta(minutes=5))
10 changes: 5 additions & 5 deletions tests/flytekit/integration/remote/workflows/basic/attr_access_dc_sd.py
View file
Original file line number Diff line number Diff line change
@@ -14,13 +14,13 @@ class DC:


@task
def create_dc() -> DC:
def create_dc(uri: str) -> DC:
"""Create a dataclass with a StructuredDataset attribute.
Returns:
dc: A dataclass with a StructuredDataset attribute.
"""
dc = DC(sd=StructuredDataset(dataframe=pd.DataFrame({"a": [5]})))
dc = DC(sd=StructuredDataset(uri=uri, file_format="parquet"))

return dc

@@ -35,10 +35,10 @@ def read_sd(dc: DC) -> StructuredDataset:


@workflow
def wf() -> None:
dc = create_dc()
def wf(uri: str) -> None:
dc = create_dc(uri=uri)
r = read_sd(dc=dc)


if __name__ == "__main__":
wf()
wf(uri="tests/flytekit/integration/remote/workflows/basic/data/df.parquet")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider configurable parquet file path

Consider making the parquet file path configurable through environment variables or configuration files instead of hardcoding it. This would make the code more flexible and easier to maintain across different environments.

Code suggestion
Check the AI-generated fix before applying
Suggested change
wf(uri="tests/flytekit/integration/remote/workflows/basic/data/df.parquet")
import os
default_path = "tests/flytekit/integration/remote/workflows/basic/data/df.parquet"
wf(uri=os.getenv("PARQUET_FILE_PATH", default_path))

Code Review Run #7e2d92

Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

Sorry, something went wrong.

Unchanged files with check annotations Beta

View file
flytekit/types/structured/structured_dataset.py
# def extract(d: Data) -> StructuredDataset:
# return d.f
if isinstance(python_val._literal_sd, StructuredDataset):
sdt = StructuredDatasetType(format=python_val._literal_sd.file_format)
metad = literals.StructuredDatasetMetadata(structured_dataset_type=sdt)
sd_literal = literals.StructuredDataset(uri=python_val._literal_sd.uri, metadata=metad)
return Literal(scalar=Scalar(structured_dataset=sd_literal))

Check warning on line 744 in flytekit/types/structured/structured_dataset.py

Codecov / codecov/patch

flytekit/types/structured/structured_dataset.py#L741-L744 

Added lines #L741 - L744 were not covered by tests
return Literal(scalar=Scalar(structured_dataset=python_val._literal_sd))