Benchmarking command benchmarks all models when max_depth is used

[danielholanda]

Issue description

Benchmarking command benchmarks all models when max_depth is used. This happens regardless of the value max_depth as long as max_depth>0.

Reproducing the issue

When running the script below using benchit model.py --analyze-only --max-depth 1 three models are discovered. However, when running benchit model.py --max-depth 1 10+ models are benchmarked.

import torch
import timm
from mlagility.parser import parse

# Creating model and set it to evaluation mode
model = timm.create_model("mobilenetv2_035", pretrained=False)
model.eval()

# Creating inputs
inputs1 = torch.rand((1, 3, 28, 28))

# Calling model
model(inputs1)

[danielholanda]

@ramkrishna2910 @jeremyfowers

[jeremyfowers]

@danielholanda can you add a bit more detail? Where are the 7 extra models coming from in the benchmarking case?

[danielholanda]

@jeremyfowers I didn't spend time analyzing this issue yet. My assumption is that those extra models corresponds to all models that would be identified if --max-depth was set to a 999.