Export workflow

[jeremyfowers]

Omnibus issue to enable the following workflow:

1. Attempt to export a large amount of PyTorch models to ONNX (Run a job to export all models #304)
2. Do not waste time on any other build steps (optimization, fp16) or benchmarking (benchit "export only" option #301 Enable any benchit option in Azure CLI #302)
3. Produce an easy-to-parse report with information about the export (e.g., the current CSV report, possibly with some extra columns TBD)
4. Save this report, in a versioned manner, somewhere accessible to the internet (see Automatically upload MLAgility reports to the internet #305 )
5. Parse the report to learn which models successfully export (see Add an API to help with parsing report.csv files #307 )

Details to follow. CC @ramkrishna2910

[jeremyfowers]

Code to parse the export status from one of our report CSV files

import pandas as pd

# Get the MLAgility report as a dataframe
df = pd.read_csv("/home/jfowers/mlagility/azure/result/2023-05-12.csv")

# Print whether each model exported successfully
print(df[["model_name", "onnx_exported"]])
print()
print()

# Convert to dictionary, access to get a single result
dc = pd.Series(df.onnx_exported.values, index=df.model_name).to_dict()

print("poolformer exported?", dc["poolformer"])

[jeremyfowers]

Code to parse the export status from one of our report CSV files

import pandas as pd

# Get the MLAgility report as a dataframe
df = pd.read_csv("/home/jfowers/mlagility/azure/result/2023-05-12.csv")

# Print whether each model exported successfully
print(df[["model_name", "onnx_exported"]])
print()
print()

# Convert to dictionary, access to get a single result
dc = pd.Series(df.onnx_exported.values, index=df.model_name).to_dict()

print("poolformer exported?", dc["poolformer"])

@ramkrishna2910 would it be helpful if this function was part of the MLAgility API?

• mlagility.api.report.get_dict(report_csv: str, column: str) -> Dict[str, str] returns a dictionary where the keys are model names and the values are the values of a given column in a report.csv file.

[ramkrishna2910]

I was originally planning to add this code on the model zoo side.
But an API on the MLAgility side to query the report makes a lot of sense. Lets add it to MLAgility.

[jeremyfowers]

• mlagility.api.report.get_dict(report_csv: str, column: str) -> Dict[str, str] returns a dictionary where the keys are model names and the values are the values of a given column in a report.csv file.

k opened #307