[Autotuner] Feature: add --cpu_budget and --timeout_per_trial

[luarss]

Rationale

• More intuitive for end users who might just want to specify the total number of cpus to be blocked.
• timeout_per_trial is different from overall timeout.

TODO

• Tests for --cpu_budget -> verify the timeout is hit.
• Tests for timeout per trial and timeout -> does it actually stop the entire training?

[oharboe]

The various ORFS stages have vastly different memory and CPU needs.

How does the user characterize and balance this?

[vvbandeira]

The various ORFS stages have vastly different memory and CPU needs.

How does the user characterize and balance this?

The intended usage for these knobs is to limit experiment runtime (and consequently $$ budget). These knobs do not limit how much resources a given ORFS run has access to.

CPU budget is intended to stop the experiment after the budget is spent.
Trial timeout will limit each ORFS complete flow runtime; this way, if your experiment requires 1k ORFS runs, you know you will, at most, use 1k * timout_per_trial units of time.