Argo for model serving

[sachinruk]

First of all, thank you to the creaters and maintainers of Hera. This has been such a godsend.

Was wondering if its possible to use argo for model serving as opposed to training/ batch jobs. It is possible to deploy a k8s app that will host a dockerised fastai endpoint (that can autoscale according to requests).

I'm hoping that given the underlying k8s architecture there is a way to make a persistent (say fastapi) endpoint with Hera. If so how would I do that?

TIA.

[agilgur5]

For reference, this was x-posted to #argo-workflows Slack, where I responded and said that while Argo can quite easily fit a batch serving model, for API-driven real-time serving, KServe/Seldon/etc are a better fit and I have used them respectively for batch vs real-time inference.

You can also use Workflows to create Deployments or InferenceServices (i.e. your MLOps pipelines), but CD may suffice for that too.

In short, there are purpose built tool stacks for each of these things, although you can certainly mix some parts together.

Also this sounds like it should've been a Discussion rather than an issue.