[Feature request]: Backend API for resource predictions

[goranbs]

Contact details

No response

Is your feature request related to a problem? Please describe

Enable predicting the scaling laws of the data pipeline using the backend API

Describe the solution you'd like

In the backend controller API, create an endpoint that takes dryRuns as input and computes the scaling parameters of input metrics. The input metrics are functions f, of the size of the input data x.

The functions f(x) are assumed to be linear (a + xb) or a power law function (ax**b)

For each step in the workflow/data pipeline, the scaling parameters are computed for each metric.

The returned data should contain something like:

output_data = {
{
step_id: string,
step_name: string,
x_data: number[],
cpu: {
y_data: number[]
scaling_law: string,
scaling_parameters: [number, number]
scaling_law_r2: number
},
...
}

where x_data is the input file size of the dryRuns and here cpu is metric data for cpu usage. The scaling law can be "power" or "linear", r2 is the coefficient of determination and the scaling parameters are coeficcients a, b for the scaling law:

linear scaling law: a + bx
power scaling law: ax**b

Describe alternatives you've considered

No response

Additional context

No response