Add image-prefix argument to mini-extra to enable use of custom docker images

[sdb9696]

Hi there. This PR adds an --image-prefix argument to the mini-extra cli so that it can be run against custom docker images. The driver for this is the desire to use the optimized epoch.ai docker images with the mini agent. Due the way they are layered the total run time for SWE-bench comes down significantly. If you are open to accepting this PR I will create a similar one for the SWE-bench repo to enable the same for evaluation runs.

Thanks

[klieret]

Hi @sdb9696 ! Really appreciate the PR and the idea.

However, I think that for custom datasets etc. you should simply have image_name set in the data source. I'm not happy that we do the __1776__ logic at all, and only have it because it's the most common input that we'll probably have. Even that I hope to remove in the future so we can be more general.

So I'd rather not merge this because it adds even more custom logic. And this kind of logic just continues to grow...

You can just download the huggingface data and set the image_name however you want and then load from a path --subset /path/to/my/dataset.

Side note: @carlosejimenez is currently updating the official swebench images to include the epoch.ai reshuffling of layers for the compression.

[klieret]

But again, really appreciate the idea!

[sdb9696]

Hi @klieret thanks for coming back. Two quick follow up questions:

• Would you be open to a PR that just took the --image-prefix param and set the image_name on instances in main? It'd be pretty small and would mean that the cli can be invoked without pre-processing.
• Is it worth creating a similar PR for for SWE-bench? afaict SWE-bench doesn't look in any config for an image_name so would need a code change to support passing custom names. cc @carlosejimenez.

[klieret]

afaict SWE-bench doesn't look in any config for an image_name so would need a code change to support passing custom names. cc @carlosejimenez.

SWE-bench v5 will definitely support that :) But this is a big refactoring that's currently underway, so i think it makes more sense to wait for that to be finished for now