[Plugin] Hamilton

[skrawcz]

What: https://github.com/stitchfix/hamilton defines a way to model a micro-workflow. Flyte models a macro view. So idea (I think) would be to run a Hamilton DAG within a flyte task more easily than it would be now.

i.e. replace this driver code that would be put into a flyte task:

import importlib

import pandas as pd
from hamilton import driver

logging.basicConfig(stream=sys.stdout)
initial_columns = {  # load from actuals or wherever -- this is our initial data we use as input.
    # Note: these do not have to be all series, they could be scalar inputs.
    'signups': pd.Series([1, 10, 50, 100, 200, 400]),
    'spend': pd.Series([10, 10, 20, 40, 40, 50]),
}
# we need to tell hamilton where to load function definitions from
module_name = 'my_functions'
module = importlib.import_module(module_name)
dr = driver.Driver(initial_columns, module)  # can pass in multiple modules
# we need to specify what we want in the final dataframe.
output_columns = [
    'spend',
    'signups',
    'avg_3wk_spend',
    'spend_per_signup',
]
# let's create the dataframe!
# if you only did `pip install sf-hamilton` earlier:
df = dr.execute(output_columns)
# else if you did `pip install sf-hamilton[visualization]` earlier:
# dr.visualize_execution(output_columns, './my-dag.dot', {})
print(df)

See https://github.com/outerbounds/hamilton-metaflow for an example of Hamilton + metaflow -- hamilton helps with the feature engineering step. So conceptually we'd try to integration simpler than just creating a task in flyte, with a plugin instead...

[welcome]

Thank you for opening your first issue here! 🛠

[skrawcz]

Related feature idea - stitchfix/hamilton#139

[skrawcz]

From @samhita-alla some examples to think look at:

For task plugins, we have:

• https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-greatexpectations/flytekitplugins/great_expectations/task.py,
• https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-papermill/flytekitplugins/papermill/task.py,

Reference documentation - https://docs.flyte.org/projects/cookbook/en/latest/auto/core/extend_flyte/custom_task_plugin.html

[github-actions]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[skrawcz]

Oops. Definitely forgot about this one. Will figure out it's priority next sprint.

[skrawcz]

So circling back to this. We're looking to schedule this for sometime in November/December.

[skrawcz]

Sorry -- still working on this one.

[skrawcz]

I'll get to this eventually...

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏

[skrawcz]

Yeah I think it makes sense to close for now. Haven't had the pull to bump it up in priority.