Moving from flyteplugins - Use dask cuda worker when GPU resource > 0

[eapolinario]

TL;DR

If a gpu has been specified in the dask plugin resource, assume we want to use dask cuda worker i.e. that dask-cuda is present on the specified image.

Type

• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

Currently we build an arg to submit to the container and inject values such as threads and memory from the limits specified in the resources object in the dask plugin object. Dask cuda gives us many built in gpu tools and all that is required is to ensure dask cuda is installed and then to start out workers with dask-cuda-worker. So, in order to accomodate, we simply check for gpu limits in the dask plugin resources and overwrite the first arg to dask-cuda-worker.

[codecov]

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (d9586b0) 58.98% compared to head (6e0886e) 57.51%.

❗ Current head 6e0886e differs from pull request most recent head be83adc. Consider uploading reports for the commit be83adc to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4129      +/-   ##
==========================================
- Coverage   58.98%   57.51%   -1.47%     
==========================================
  Files         619      394     -225     
  Lines       52804    28988   -23816     
==========================================
- Hits        31146    16673   -14473     
+ Misses      19174    10626    -8548     
+ Partials     2484     1689     -795     

Flag	Coverage Δ
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

see 575 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.