What does this PR do?

This PR does a few things:

• Fixes reproducibility issues in skythought. The core issue is performing inference in
  half precision. More details to follow. We now use float32 by default.
• Adds support for saving metrics to disk (along with token usage statistics).
• Adds support for computing pass@k for args.n > 1. Automatically computes pass@k for different powers of 2 less than args.n
• Cleans up minor bugs after merging [evals] Add support for scaling evals and inference with ray #63 and Refactor model-specific configs and move data curation scripts #60.
• Removes reading stdout output in eval.py. This is extremely costly for large datasets or for args.n > 1 because we simply iterate over all the logs in a for loop. But default, child process logs get streamed to the stdout of the parent process. If we want to save all logs to disk, we can just use tee . Saving metrics explicitly also eliminates this.

New Metrics File: Example

{
    "completion_tokens": 4265446,
    "prompt_tokens": 176656,
    "avg_completion_tokens": 8886.346,
    "avg_prompt_tokens": 368.033,
    "pass_at_k": {
        "temp=0.7": {
            "k=16": 66.667,
            "k=8": 60.569,
            "k=4": 52.879,
            "k=2": 44.028,
            "k=1": 35.417
        }
    },
    "accuracy": {
        "temp=0.7": 0.3542
    }
}

TODO:

• Re-compute AIME and GPQA Diamond scores
• Improve pass@k calculation

Should fix: #66, #48