Hash model inputs instead of parameters

[danielholanda]

Closes #322

Overview

This PR stops hashing the models using weights and hashes input shapes instead.

Description

This PR closes #322 by implement the long-term solution and revert the workaround mentioned here: #316 (comment) (closing #316 in the same PR).

It also adds a test to test/analysis.py to prevent this kind of break in the future.

Manually testing

CI tests were added. However, you may manually test this PR by running benchit two_calls.py --analyze-only.

two_calls.py

# labels: name::mobilenetv2_035 author::timm task::Computer_Vision
import torch
import timm
from mlagility.parser import parse

# Parsing command-line arguments
batch_size = parse(["batch_size"])

# Creating model and set it to evaluation mode
model = timm.create_model("mobilenetv2_035", pretrained=False)
model.eval()

# Creating inputs
inputs1 = torch.rand((1, 3, 28, 28))
inputs2 = torch.rand((1, 3, 224, 224))

# Calling model
model(inputs1)
model(inputs2)

Expected results:

Models discovered during profiling:

two_input_shapes.py:
        model (executed 1x - 0.01s)
                Model Type:     Pytorch (torch.nn.Module)
                Class:          EfficientNet (<class 'timm.models.efficientnet.EfficientNet'>)
                Location:       /home/dhnoronha/.local/lib/python3.8/site-packages/timm/models/efficientnet_builder.py, line 472
                Parameters:     1,677,128 (3.2 MB)
                Input Shape:    {'Positional Arg 1': (1, 3, 28, 28)}
                Hash:           b63f3042

        model (executed 1x - 0.03s)
                Model Type:     Pytorch (torch.nn.Module)
                Class:          EfficientNet (<class 'timm.models.efficientnet.EfficientNet'>)
                Location:       /home/dhnoronha/.local/lib/python3.8/site-packages/timm/models/efficientnet_builder.py, line 472
                Parameters:     1,677,128 (3.2 MB)
                Input Shape:    {'Positional Arg 1': (1, 3, 224, 224)}
                Hash:           062ae984


Woohoo! The 'benchmark' command is complete.

Note: Input Shape is only displayed when a model has more than one workload

[danielholanda]

Should we start differentiating between workloads and models in our documentation?

Examples:

• Replace Each model in a script is identified by a unique hash with Each workload in a script is identified by a unique hash. (user guide)
• Replace Models discovered during profiling: with Workloads discovered during profiling: (cli message)

[jeremyfowers]

Discuss architecture

[danielholanda]

Heads up: Significant changes were made to ensure that analysis also works when we have the same models with multiple inputs AND max-depth is > 1. Here is one example of things working correctly.

Note that the shapes and hashes of the "deeper" models are different when you compare the two workloads.

two_input_shapes.py:
        model (executed 1x - 0.18s)
                Model Type:     Pytorch (torch.nn.Module)
                Class:          EfficientNet (<class 'timm.models.efficientnet.EfficientNet'>)
                Location:       /home/dhnoronha/.local/lib/python3.8/site-packages/timm/models/efficientnet_builder.py, line 472
                Parameters:     1,677,128 (3.2 MB)
                Input Shape:    'Positional Arg 1': (1, 3, 28, 28)
                Hash:           4c4868c4

                        blocks (executed 1x - 0.17s)
                                Model Type:     Pytorch (torch.nn.Module)
                                Class:          Sequential (<class 'torch.nn.modules.container.Sequential'>)
                                Parameters:     249,744 (0.5 MB)
                                Input Shape:    'Positional Arg 1': (1, 16, 14, 14)
                                Hash:           2b4daad8

                        bn2 (executed 1x - 0.00s)
                                Model Type:     Pytorch (torch.nn.Module)
                                Class:          BatchNormAct2d (<class 'timm.models.layers.norm_act.BatchNormAct2d'>)
                                Parameters:     2,560 (<0.1 MB)
                                Input Shape:    'Positional Arg 1': (1, 1280, 1, 1)
                                Hash:           26e367bf

        model (executed 1x - 0.22s)
                Model Type:     Pytorch (torch.nn.Module)
                Class:          EfficientNet (<class 'timm.models.efficientnet.EfficientNet'>)
                Location:       /home/dhnoronha/.local/lib/python3.8/site-packages/timm/models/efficientnet_builder.py, line 472
                Parameters:     1,677,128 (3.2 MB)
                Input Shape:    'Positional Arg 1': (1, 3, 224, 224)
                Hash:           cbdf10c5

                        blocks (executed 1x - 0.21s)
                                Model Type:     Pytorch (torch.nn.Module)
                                Class:          Sequential (<class 'torch.nn.modules.container.Sequential'>)
                                Parameters:     249,744 (0.5 MB)
                                Input Shape:    'Positional Arg 1': (1, 16, 112, 112)
                                Hash:           7714ae20

                        bn2 (executed 1x - 0.00s)
                                Model Type:     Pytorch (torch.nn.Module)
                                Class:          BatchNormAct2d (<class 'timm.models.layers.norm_act.BatchNormAct2d'>)
                                Parameters:     2,560 (<0.1 MB)
                                Input Shape:    'Positional Arg 1': (1, 1280, 7, 7)
                                Hash:           83405a1c

[danielholanda]

@jeremyfowers I converted this PR back to draft since I intend to change the messages displayed to the user as discussed in #324 (comment)

[jeremyfowers]

Looking great, this is pretty much good to go! Just doing "request changes" so that I have a chance to check the documentation addition before this merges.

[jeremyfowers]

Thank you for all the brainstorming and multiple rounds of feedback! This change wound up in a really good place.

I pushed a few commits with minor copy-editing and I added your new example to CI (no functionality changes).