Separate time partition #2114

wild-endeavor · 2024-01-18T02:16:51Z

This uses the separate time partition message added in flyteorg/flyte#4737.

Signed-off-by: Thomas J. Fan <[email protected]>

…hat take dataclasses (#2013) Signed-off-by: jason.lai <[email protected]>

Signed-off-by: troychiu <[email protected]>

* Error when multiple configuration plugins are installed Signed-off-by: Thomas J. Fan <[email protected]> * Adds semi-colon Signed-off-by: Thomas J. Fan <[email protected]> --------- Signed-off-by: Thomas J. Fan <[email protected]>

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Signed-off-by: pryce-turner <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Hongxin Liang <[email protected]>

Signed-off-by: nikki everett <[email protected]>

* added _cmd_prefix handling Signed-off-by: Daniel Rammer <[email protected]> * fixed typing imports Signed-off-by: Daniel Rammer <[email protected]> * add get_config Signed-off-by: Kevin Su <[email protected]> * updating get_config to use underlying functions config Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Kevin Su <[email protected]>

Signed-off-by: DouEnergy <[email protected]>

Signed-off-by: Austin Liu <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>

* Accelerator docs Signed-off-by: Ketan Umare <[email protected]> * updating docs Signed-off-by: Ketan Umare <[email protected]> * Updated docs Signed-off-by: Ketan Umare <[email protected]> * updated docs Signed-off-by: Ketan Umare <[email protected]> * more docs update Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated docs Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Co-authored-by: Ketan Umare <[email protected]>

* Add support for running remote-tasks and remote-workflows Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]> Co-authored-by: Ketan Umare <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

Bumps [fonttools](https://github.com/fonttools/fonttools) from 4.41.1 to 4.43.0. - [Release notes](https://github.com/fonttools/fonttools/releases) - [Changelog](https://github.com/fonttools/fonttools/blob/main/NEWS.rst) - [Commits](fonttools/fonttools@4.41.1...4.43.0) --- updated-dependencies: - dependency-name: fonttools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* add monodocs index page in flytekit Signed-off-by: Niels Bantilan <[email protected]> * updates Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]>

* add monodocs redirects to the flytekit readthedocs project Signed-off-by: Niels Bantilan <[email protected]> * update requirements Signed-off-by: Niels Bantilan <[email protected]> * add redirects enabling flag Signed-off-by: Niels Bantilan <[email protected]> * update Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

) Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.2...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Signed-off-by: Thomas J. Fan <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* Use Latest FlyteIdl in all Dockerfile Signed-off-by: Future Outlier <[email protected]> * lint Signed-off-by: Future Outlier <[email protected]> * Revert all dockerfile except dev version Signed-off-by: Future-Outlier <[email protected]> * remove sqlalchemy dockerfile Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future Outlier <[email protected]> Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>

* Fix typos and run spellcheck in CI Signed-off-by: Eduardo Apolinario <[email protected]> * Remove --interactive Signed-off-by: Eduardo Apolinario <[email protected]> * Remove CI check since it's run in pre-commit checks (lint make target) Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> * Fix typo in accelerators.py Signed-off-by: Eduardo Apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

…ct is specified correctly, raise unbound errors, fix tests Signed-off-by: Yee Hing Tong <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>

codecov · 2024-01-19T22:50:05Z

Codecov Report

Attention: 660 lines in your changes are missing coverage. Please review.

Comparison is base (d3e1884) 78.73% compared to head (77004bd) 53.54%.
Report is 53 commits behind head on artifacts.

Files	Patch %	Lines
flytekit/core/artifact.py	21.36%	265 Missing ⚠️
flytekit/trigger.py	0.00%	79 Missing ⚠️
flytekit/remote/remote.py	15.85%	67 Missing and 2 partials ⚠️
flytekit/core/type_engine.py	5.12%	37 Missing ⚠️
flytekit/types/directory/types.py	8.57%	32 Missing ⚠️
flytekit/image_spec/image_spec.py	29.03%	22 Missing ⚠️
flytekit/extend/backend/base_agent.py	37.93%	16 Missing and 2 partials ⚠️
flytekit/loggers.py	67.27%	8 Missing and 10 partials ⚠️
flytekit/core/interface.py	51.51%	10 Missing and 6 partials ⚠️
flytekit/core/workflow.py	53.33%	13 Missing and 1 partial ⚠️
... and 26 more

Additional details and impacted files

@@              Coverage Diff               @@
##           artifacts    #2114       +/-   ##
==============================================
- Coverage      78.73%   53.54%   -25.20%     
==============================================
  Files            173      178        +5     
  Lines          16815    17582      +767     
  Branches        2310     3650     +1340     
==============================================
- Hits           13240     9414     -3826     
- Misses          3004     7737     +4733     
+ Partials         571      431      -140

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

* temp Signed-off-by: Yee Hing Tong <[email protected]> * stuff Signed-off-by: Yee Hing Tong <[email protected]> * temp Signed-off-by: Yee Hing Tong <[email protected]> * scaffolding areas mostly identified Signed-off-by: Yee Hing Tong <[email protected]> * add artifact to upload request Signed-off-by: Yee Hing Tong <[email protected]> * remove an unnecessary line in workflow Signed-off-by: Yee Hing Tong <[email protected]> * finish adding examples use cases maybe Signed-off-by: Yee Hing Tong <[email protected]> * add project/dom to get query Signed-off-by: Yee Hing Tong <[email protected]> * add from flyte idl Signed-off-by: Yee Hing Tong <[email protected]> * add project domain to as query Signed-off-by: Yee Hing Tong <[email protected]> * add condition in parameter to flyte idl Signed-off-by: Yee Hing Tong <[email protected]> * test stuff * Remove artifactID from literal oneof, add to metadata (#2) * Triggers (#6) * Minor changes to get time series example working #8 Signed-off-by: Yee Hing Tong <[email protected]> * switch channel (#10) Signed-off-by: Yee Hing Tong <[email protected]> * fix tests ignore - pr into other pr (#1858) Signed-off-by: Yee Hing Tong <[email protected]> * Artf/update idl ux (#1920) Signed-off-by: Yee Hing Tong <[email protected]> * Artf/trigger (#1948) * Add triggers * Remove bind_partition_time and just assume users won't use that. It's just time_partition in the normal call function now. Signed-off-by: Yee Hing Tong <[email protected]> * remove the now deleted artifact spec (#1984) Signed-off-by: Yee Hing Tong <[email protected]> * Literal metadata model update (#2089) Signed-off-by: Yee Hing Tong <[email protected]> * Separate time partition (#2114) Signed-off-by: Yee Hing Tong <[email protected]> * Split service code (#2136) Signed-off-by: Yee Hing Tong <[email protected]> * remove empty files Signed-off-by: Yee Hing Tong <[email protected]> * add noneness check to metadata and add test Signed-off-by: Yee Hing Tong <[email protected]> * remove sandbox test for now Signed-off-by: Yee Hing Tong <[email protected]> * Artf/cleanup (#2158) * add a test Signed-off-by: Yee Hing Tong <[email protected]> * try updates Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> * Use python 3.9 to run make doc-requirements.txt Signed-off-by: Eduardo Apolinario <[email protected]> * reasons not msg Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>

* temp Signed-off-by: Yee Hing Tong <[email protected]> * stuff Signed-off-by: Yee Hing Tong <[email protected]> * temp Signed-off-by: Yee Hing Tong <[email protected]> * scaffolding areas mostly identified Signed-off-by: Yee Hing Tong <[email protected]> * add artifact to upload request Signed-off-by: Yee Hing Tong <[email protected]> * remove an unnecessary line in workflow Signed-off-by: Yee Hing Tong <[email protected]> * finish adding examples use cases maybe Signed-off-by: Yee Hing Tong <[email protected]> * add project/dom to get query Signed-off-by: Yee Hing Tong <[email protected]> * add from flyte idl Signed-off-by: Yee Hing Tong <[email protected]> * add project domain to as query Signed-off-by: Yee Hing Tong <[email protected]> * add condition in parameter to flyte idl Signed-off-by: Yee Hing Tong <[email protected]> * test stuff * Remove artifactID from literal oneof, add to metadata (#2) * Triggers (#6) * Minor changes to get time series example working #8 Signed-off-by: Yee Hing Tong <[email protected]> * switch channel (#10) Signed-off-by: Yee Hing Tong <[email protected]> * fix tests ignore - pr into other pr (#1858) Signed-off-by: Yee Hing Tong <[email protected]> * Artf/update idl ux (#1920) Signed-off-by: Yee Hing Tong <[email protected]> * Artf/trigger (#1948) * Add triggers * Remove bind_partition_time and just assume users won't use that. It's just time_partition in the normal call function now. Signed-off-by: Yee Hing Tong <[email protected]> * remove the now deleted artifact spec (#1984) Signed-off-by: Yee Hing Tong <[email protected]> * Literal metadata model update (#2089) Signed-off-by: Yee Hing Tong <[email protected]> * Separate time partition (#2114) Signed-off-by: Yee Hing Tong <[email protected]> * Split service code (#2136) Signed-off-by: Yee Hing Tong <[email protected]> * remove empty files Signed-off-by: Yee Hing Tong <[email protected]> * add noneness check to metadata and add test Signed-off-by: Yee Hing Tong <[email protected]> * remove sandbox test for now Signed-off-by: Yee Hing Tong <[email protected]> * Artf/cleanup (#2158) * add a test Signed-off-by: Yee Hing Tong <[email protected]> * try updates Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> * Use python 3.9 to run make doc-requirements.txt Signed-off-by: Eduardo Apolinario <[email protected]> * reasons not msg Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]>

thomasjpfan and others added 30 commits December 22, 2023 13:17

Allows Secret groups to be optional and configurable (#2062)

68430db

Signed-off-by: Thomas J. Fan <[email protected]>

[Core feature] Allow dictionaries to be passed to execute workflows t…

eab12e6

…hat take dataclasses (#2013) Signed-off-by: jason.lai <[email protected]>

fix context usage (#2070)

0e8abd9

Signed-off-by: troychiu <[email protected]>

test (#2077)

452a2ec

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Generalized subproc_execute and added tests (#2072)

d1ce05b

Signed-off-by: pryce-turner <[email protected]>

Improve error handling for local execution (#2068)

76ebae2

Signed-off-by: Kevin Su <[email protected]>

Download specific files in FlyteDirectory (#2059)

6f613e7

Signed-off-by: Kevin Su <[email protected]>

Constraint the full fsspec family (#2087)

9bf7afa

Signed-off-by: Hongxin Liang <[email protected]>

Update default template used by pyflyte init (#2041)

5841a1e

Signed-off-by: nikki everett <[email protected]>

remove unnecessary newline in plugin readme (#2090)

9c45cb2

Signed-off-by: DouEnergy <[email protected]>

add fs path exists checking (fix flyteorg/flyte#4470) (#2079)

f22cec2

Signed-off-by: Austin Liu <[email protected]>

add monodocs redirect banner (#2091)

b5dbd85

Signed-off-by: Niels Bantilan <[email protected]>

Print log links in local execution (#2097)

dc35e7f

Signed-off-by: Kevin Su <[email protected]>

add monodocs index page in flytekit (#2100)

d451ee2

* add monodocs index page in flytekit Signed-off-by: Niels Bantilan <[email protected]> * updates Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]>

Local Sync Agent (#2085)

ed91dd1

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Enhance Airflow Plugin (#2099)

76f6db2

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

init (#2078)

a3a640a

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Fix Flyin Plugin VSCode download bug (#1991)

ccc515e

Signed-off-by: Future Outlier <[email protected]> Co-authored-by: Future Outlier <[email protected]>

Adds conda support to envd plugin (#2020)

892b474

Signed-off-by: Thomas J. Fan <[email protected]>

Fix is_deferrable in airflow agent (#2109)

80ca660

Signed-off-by: Kevin Su <[email protected]>

Future-Outlier and others added 3 commits January 17, 2024 18:19

Changes default destination dir to be "working dir" (#2108)

7996c2e

use separate time partition in idl change flyteorg/flyte#4737

692b78c

Signed-off-by: Yee Hing Tong <[email protected]>

wild-endeavor requested review from kumare3, eapolinario, pingsutw and cosmicBboy as code owners January 18, 2024 02:16

wild-endeavor and others added 9 commits January 17, 2024 18:17

nit

0e66bc4

Signed-off-by: Yee Hing Tong <[email protected]>

remove the constant as it's no longer needed

2f85ed0

Signed-off-by: Yee Hing Tong <[email protected]>

Add @lru_cache to ImageBuildEngine.build (#2112)

6279b81

Signed-off-by: Kevin Su <[email protected]>

remove tag from detect_artifact, handle the case where just an artifa…

4b232f1

…ct is specified correctly, raise unbound errors, fix tests Signed-off-by: Yee Hing Tong <[email protected]>

try adding git path for pip

7677417

Signed-off-by: Yee Hing Tong <[email protected]>

Merge remote-tracking branch 'origin/master' into artf/separate-time

ce58a8d

Signed-off-by: Yee Hing Tong <[email protected]>

add clarifying comment that came up in testing

78d22e3

Signed-off-by: Yee Hing Tong <[email protected]>

revert

77004bd

Signed-off-by: Yee Hing Tong <[email protected]>

wild-endeavor merged commit 63b7b29 into artifacts Jan 24, 2024
10 of 25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Separate time partition #2114

Separate time partition #2114

wild-endeavor commented Jan 18, 2024

codecov bot commented Jan 19, 2024 •

edited

Loading

Separate time partition #2114

Separate time partition #2114

Conversation

wild-endeavor commented Jan 18, 2024

codecov bot commented Jan 19, 2024 • edited Loading

Codecov Report

codecov bot commented Jan 19, 2024 •

edited

Loading