Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add kubernetes template for airflow + more #120

Merged
merged 41 commits into from
Jul 21, 2024
Merged

Add kubernetes template for airflow + more #120

merged 41 commits into from
Jul 21, 2024

Conversation

arthurprevot
Copy link
Owner

@arthurprevot arthurprevot commented Jul 15, 2024 •
edited
Loading

Added ability to create airflow dag for kubernetes job. It is triggered by new deploy option --deploy=airflow_k8s. Related:

  • creating new template for k8s airflow jobs.
  • made dag local folder parametrizable (i.e. hardcoded to tmp/ folder) and upload optional + added unit-tests (checking reference dag files) + new test job to showcase it.

Other:

  • renamed expand_later() to expand_latest()
  • added env_dispatchers / list_files() and env_dispatchers / copy_files()
  • fixed yml_mode selection and defaults
  • improved AWS connection (cutting and pasting creds in terminal now work for all AWS connections), and added test_aws_connection()
  • fixed issue with setup_master.sh, better logging

Moved to 0.12.1 to integrate this PR

@arthurprevot
make deploy to k8s pre-parametrized for k8s_aws or for k8s_docker_des…
93a5467 
…ktop + unit test.
@arthurprevot
function rename path_handler().expand_later() -> expand_latest()
907dbc9
@arthurprevot
added list_files() and copy_files() for cross plateform ops
5f5cfdb
@arthurprevot
update setup_master.sh to diagnose update issue in EMR ( with aws s3 …
e53d55d 
…cp command not doing the expected transfer)
@arthurprevot
cleaning
04652d8
@arthurprevot
deal with default_aws_modes from yml file
7b243b1
@arthurprevot
get deploy.py to use etl_utils.get_aws_setup() to get aws creds from …
f88ee50 
…env if available.
@arthurprevot
added --deploy=airflow_k8s that uses a new template (not yet set, pla…
0f817f7 
…ceholder for now)
@arthurprevot
add template for airflow_k8s, just copy of emr for now. TBUpdated
b97aea0
@arthurprevot
customize airflow_template_k8s.py for k8s
ea35706
@arthurprevot
customize
e702c27
@arthurprevot
template yaml for spark submit for airflow. to be customized later.
8b53d87
@arthurprevot
set k8s params for template
9a00046
@arthurprevot
added k8s params in jobs_metadata.yml
37ea13e
@arthurprevot
fix mode selection + fix jargs
8df6167
@arthurprevot
var rename
8159b1e
@arthurprevot
remove code now not needed
abf7fd6
@arthurprevot
make dag local folder parametrizable and upload optional + added unit…
c4eea4d 
…-test + new test job to showcase it.
@arthurprevot
fix get_aws_setup()
82e4c3b
@arthurprevot arthurprevot changed the title Clean dev emr Add kubernetes template for airflow + more Jul 15, 2024
@arthurprevot arthurprevot merged commit 2880f54 into master Jul 21, 2024
2 checks passed
@arthurprevot arthurprevot deleted the clean_dev_emr branch July 21, 2024 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@arthurprevot