action tracing for anomaly detection

[dragonstyle]

This PR introduces a new trace logging mode that is intended to detect actions which do not complete (and thus could be the cause of a run hanging). We'll be making significant enhancements to this soon (e.g. incorporating into the task UI) but the basic version implemented here is useful as a dianostic tool.

There are several things which may cause a task to not complete:

1. A generate() call to a model does not return.
2. A call to subprocess() without a timeout does not return.
3. An interaction with Docker containers or the Docker daemon does not terminate.
4. Writing to remote storage (e.g. S3) does not return.
5. A tool call does not return (some sort of infinite loop)
6. A subtask does not complete (again, some soft of infinite loop)

Trace logs aim to detect these by writing logs for every evaluation that record the beginning and end of each action (as well as cancellations and errors). Trace logs for the last 10 evaluations are preserved.

If an evaluation is running and is not terminating, you can execute the following command to list instances of the above actions (e.g. generate, docker, tool calls, etc.) that are still running:

inspect trace anomalies

You first see currently running actions (useful mostly for a "live" evaluation). If you have already cancelled an evaluation you'll see a list of cancelled actions (with the most recently completed cancelled action on top) which will often also tell you which unterminated action was keeping an evaluation from completing.

There will also be a list of actions that ended in an error. Note that actions which are cancelled (e.g. due to timeouts) or have errors (e.g. due to model API errors) aren't necessarily a problem as some timeouts and error conditions naturally occur in evaluations. They are there for additional context but are not the central purpose of inspect trace anomolies (which is to detect actions that unexpectedly fail to terminate).

You can also look at anomalies for a specific trace log (as opposed to the most recent one) as follows:

inspect trace anomalies trace.log

If you want to explore the trace files directly, you can list their paths with:

inspect trace list # --json for JSON output

Trace logs are in JSON lines, you can read them as a JSON array with:

inspect trace read trace.log