Hybrid Parallel Plugin下TP显存比同配置下deepspeed要高？？？

[duomicoding]

您好，请问下为啥Hybrid Parallel Plugin下TP显存比同配置下deepspeed要高？？？

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: Is the TP memory under Hybrid Parallel Plugin higher than the deepspeed under the same configuration? ? ?

Hello, may I ask why the TP memory under Hybrid Parallel Plugin is higher than the deepspeed under the same configuration? ? ?

[duomicoding]

好像是显存碎片造成？而且很严重，请问下有对应的优化改进措施吗？

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

It seems to be caused by memory fragmentation? And it’s very serious. Are there any corresponding optimization and improvement measures?