Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SAM2 to TorchBench model set #2566

Open
huydhn opened this issue Jan 16, 2025 · 4 comments
Open

Add SAM2 to TorchBench model set #2566

huydhn opened this issue Jan 16, 2025 · 4 comments
Assignees
@huydhn

Comments

@huydhn
Copy link
Contributor

huydhn commented Jan 16, 2025

We want to add SAM2 from https://github.com/facebookresearch/sam2?tab=readme-ov-file#download-checkpoints to TorchBench so that it can be included as part of PT2 OSS benchmark run.

cc @kit1980 Let's take this opportunity to follow @xuzhao9 documentation in https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/ADDING_MODELS.md to add the model.

cc @cpuhrsch
@huydhn huydhn self-assigned this Jan 16, 2025
@cpuhrsch
Copy link
Contributor

cc @josephgreer

@cpuhrsch
Copy link
Contributor

cpuhrsch commented Jan 16, 2025 •
edited
Loading

In particular we'd need the backbone models with the various configurations checkpoints loaded via

https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L472-L480

It's sufficient to use the resulting model for a particular task to exercise this. For example https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L294 on some example shape such as https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L63 after instantiating an instance via
https://github.com/pytorch/ao/blob/522f5b854a278ee9e68e80bf8213e19c9da4e547/examples/sam2_amg_server/server.py#L478-L480

@malfet
Copy link
Contributor

malfet commented Jan 21, 2025 •
edited
Loading

Should we try to add it somehow directly to PyTorch instead? As CI pin is pretty much behind repo head.

@ZainRizvi ZainRizvi moved this to Cold Storage in PyTorch OSS Dev Infra Jan 21, 2025
@yangw-dev
Copy link

yangw-dev commented Jan 22, 2025 •
edited
Loading

[can be deleted][just to myself]
what is PT2 OSS benchmark run

@huydhn huydhn mentioned this issue Jan 29, 2025
Closed
facebook-github-bot pushed a commit that referenced this issue Jan 29, 2025
@huydhn @facebook-github-bot
Update torchbench pin on PyTorch CI (#2584)
0e370a0 
Summary:
I'm adding sam2 to TorchBench #2566, so I'm updating PyTorch CI to use latest TorchBench commit.

This goes together with pytorch/pytorch#145455.  The list of fixes includes:

* Adding `decoder_start_token_id` to HF `GenerationConfig`. This was Introduced by transformers 4.41.0 huggingface/transformers#30892
* Run `sam_fast` on A10G as it seems working correctly now

Pull Request resolved: #2584

Reviewed By: xuzhao9

Differential Revision: D68837682

Pulled By: huydhn

fbshipit-source-id: d9bda87384bac680b08fd2ea200ca7691c24a7e9
@huydhn huydhn mentioned this issue Jan 30, 2025
Closed
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this issue Feb 1, 2025
@huydhn @pytorchmergebot
Update TorchBench commit to main (#145455)
f38d5b4 
I'm adding sam2 to TorchBench pytorch/benchmark#2566, so, as part of that, I'm updating PyTorch CI to use latest TorchBench commit.

The corresponding change from TorchBench is pytorch/benchmark#2584

The main thing to call out that the newer transformers added by pytorch/benchmark#2488 is regressing several models. This needs to be investigated further, and I pin the version to unblock this change.

* `hf_Roberta_base` a new model added by pytorch/benchmark#2279, not sure why it fails accuracy on A10G, but it works fine on A100
* `speech_transformer` failures are pre-existing trunk failures, i.e. https://github.com/pytorch/pytorch/actions/runs/13040114684/job/36380989702#step:22:2408

Pull Request resolved: #145455
Approved by: https://github.com/kit1980
facebook-github-bot pushed a commit that referenced this issue Feb 3, 2025
@huydhn @facebook-github-bot
Update TorchBench commit to main (#145455)
76d4920 
Summary:
I'm adding sam2 to TorchBench #2566, so, as part of that, I'm updating PyTorch CI to use latest TorchBench commit.

The corresponding change from TorchBench is #2584

The main thing to call out that the newer transformers added by #2488 is regressing several models. This needs to be investigated further, and I pin the version to unblock this change.

* `hf_Roberta_base` a new model added by #2279, not sure why it fails accuracy on A10G, but it works fine on A100
* `speech_transformer` failures are pre-existing trunk failures, i.e. https://github.com/pytorch/pytorch/actions/runs/13040114684/job/36380989702#step:22:2408

X-link: pytorch/pytorch#145455
Approved by: https://github.com/kit1980

Reviewed By: ZainRizvi

Differential Revision: D69056903

fbshipit-source-id: 6b98c246b9dc8811257ec46d4821ba6ee3363dfe
mori360 pushed a commit to mori360/pytorch that referenced this issue Feb 6, 2025
@huydhn @mori360
Update TorchBench commit to main (pytorch#145455)
daffd89 
I'm adding sam2 to TorchBench pytorch/benchmark#2566, so, as part of that, I'm updating PyTorch CI to use latest TorchBench commit.

The corresponding change from TorchBench is pytorch/benchmark#2584

The main thing to call out that the newer transformers added by pytorch/benchmark#2488 is regressing several models. This needs to be investigated further, and I pin the version to unblock this change.

* `hf_Roberta_base` a new model added by pytorch/benchmark#2279, not sure why it fails accuracy on A10G, but it works fine on A100
* `speech_transformer` failures are pre-existing trunk failures, i.e. https://github.com/pytorch/pytorch/actions/runs/13040114684/job/36380989702#step:22:2408

Pull Request resolved: pytorch#145455
Approved by: https://github.com/kit1980
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@huydhn huydhn

Labels
None yet
Projects
PyTorch OSS Dev Infra
Status: Cold Storage
Milestone
No milestone
Development

No branches or pull requests
4 participants
@huydhn @cpuhrsch @malfet @yangw-dev