text encoder(google/mt5-xxl) can be substituted by a smaller one?

[hongfengvvv]

The file size of text encoder(google/mt5-xxl) model is more than 50GB. Is it possible to replace this model with a smaller one?
Or, will there be a smaller version of this model available in the future?

[sguo112]

You can try mt5-small

[huipengzhang]

When I test Open-Sora-Plan1.3 use mt5-small ,it will make a error for this

I Identified the hidden_size of mt5-small is 512. It may causing shape mismatch between input and weight.How should we use mt5-small to inference,and do you need to modify any configurations? @sguo112

[sguo112]

I haven't succeeded yet@huipengzhang

[yunyangge]

Hello, by default, smaller versions of mT5 cannot be used. Because their output dimensions are different from mT5-XXL, this results in a mismatch between DiT's weight dimensions and theirs.