Skip to content

TuringLang/ADTests

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

93 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ADTests

This repository is used to tabulate the current performance of various automatic differentiation (AD) backends with Turing.jl models.

Let me see the results!

https://turinglang.org/ADTests

I want to add more AD backends!

You can modify the list of AD types in main.jl.

I want to add more models!

You can modify the list of models in models.jl.

I want to edit the HTML that gets generated!

The HTML is generated by the ad.py script. You can edit it there.

Note that the links-to-existing-GitHub-issues in the table are also defined in this script.

I want to see the HTML generated by a PR!

The latest workflow run across all PRs will be published to https://turinglang.org/ADTests/pr.

This is a bit messy, but works for now on the assumption that there aren't many PRs being worked on simultaneously.

What's going on?

The workflow is the most complicated part of this repository. This section attempts to explain it from the 'bottom up'; if you prefer a 'top down' approach start by looking at the GitHub Action workflow, .github/workflows/test.yml.

Under the hood, the main thing that actually runs the AD tests / benchmarks is main.jl. You can run julia --project=. main.jl and it will print some usage information. However, it is the Python script ad.py that controls how this Julia script is called.

Fundamentally, the idea is that we want to loop over every combination of model and adtype and test it. However, because GitHub Actions limits jobs to 6 hours, it is impractical to run every combination in the same job. What we do is to run one job per model. This is accomplished by first storing the names of the models and adtypes in the $GITHUB_OUTPUT variable using python ad.py setup, which can then be read by the next job.

The next job is a CI matrix split by model name; each of the sub-jobs invokes python ad.py run --model {model_name}, which loops over each adtype and calls the Julia script with the model name and adtype as arguments. The purpose of having this Python -> Julia setup (as opposed to doing the looping inside Julia itself) is to guard against the Julia process crashing, which can happen sporadically with Enzyme. If the Julia process successfully finishes, it will print the result which is picked up by the Python script; if it crashes, we just record the result as 'error'.

Finally, the results are collated and sent to the final job in the workflow, which is python ad.py html. It could be written in Julia (PRs are welcome). However, it was much faster for me to do it in Python. (Fun fact: collating these results is also somewhat involved because we can't just write to $GITHUB_OUTPUT; it turns out that output from different jobs in a matrix will override each other. Thankfully, there is an existing action which is designed to get around this problem by uploading artefacts.)

Overall, what this means is that the entire table can be generated in around 10 minutes (longer if you need to install + precompile dependencies, but on GitHub Actions dependencies will for the most part have been cached).

Can I run this locally?

Yes, but it's quite tricky.

The easiest thing to do is to run one specific combination of model and adtype. This doesn't require any environment variables: you can run julia --project=. main.jl --run <model> <adtype>.

However, if you want to run the Python script locally, you will need to set the RESULTS_JSON environment variable. The easiest way to do this is to go to the GitHub Actions log for the collate job, where the value of RESULTS_JSON is printed. It is a multiline string (it's JSON), so you will need to do something like this:

read -d '' RESULTS_JSON << EOF
(paste the JSON here)
EOF

export RESULTS_JSON

Then you can run the Python script locally with python ad.py html. It will generate the HTML file in the html directory.

PRs to make this simpler are more than welcome.

Releases

No releases published

Packages

No packages published