Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize tp broadcast #2889

Merged
merged 62 commits into from
Dec 17, 2024
Merged

Optimize tp broadcast #2889

merged 62 commits into from
Dec 17, 2024
+149 −17

Conversation

grimoire
Copy link
Collaborator

@grimoire grimoire commented Dec 12, 2024 •
edited
Loading

lvhan028 and others added 30 commits November 18, 2024 21:28
@lvhan028
refactor VL modules for internvl and qwen2-vl (InternLM#2764)
464d451 
* qwen2-vl

* internvl

* qwen2
@lvhan028
Refactor VL modules for glm4v, deepseek-vl, llava-hf, cogvlm (InternL…
3ba8309 
…M#2772)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava
@lvhan028
Refactor VL modules for qwen-vl, llava and llava_next (InternLM#2773)
accfc00 
* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava
@lvhan028
Refactor VL modules for qwen2-vl (InternLM#2777)
463b508 
* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2
@lvhan028
Fix side-effect to internvl (InternLM#2778)
9e18529 
* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl
@lvhan028
Refactor VL modules for phi3-vision (InternLM#2779)
b2c29be 
* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* phi3-vision
@lvhan028
Refactor VL modules for mllama and yi-vl (InternLM#2781)
aca0f2d 
* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* phi3-vision

* refactor yi-vl

* refactor mllama
@lvhan028
Refactor VLM module for minicpm and molmo (InternLM#2794)
fff7b91
@lvhan028
Refactor VLM modules for xcomposer series (InternLM#2796)
45cf22d
@lvhan028
Refactor VLM modules for internvl-llava (InternLM#2797)
38eec0d
@lvhan028
Refactor VLM modules v2 (InternLM#2806)
c4638d5 
* internvl2 v2

* cogvlm

* deepseek-vl

* glm-4v

* llava-hf

* llava-next

* llava

* internvl-llava

* mllama

* phi3-vision

* qwen

* qwen2

* yi-vl

* xcomposer

* minicpm

* molmo

* update

* update
@lvhan028
Remove vl template (InternLM#2809)
b07211b
@lvhan028 @akai-shuuichi
@jinminxi104 @zhulinJulia24 @Reinerzhou @AllentDan @JackWeiw
Resolve conflicts (InternLM#2811)
e7b5a7e 
* feature: support qwen2.5 fuction_call (InternLM#2737)

* feat: support qwen2.5 tools_call

* fix: npe bug

* fix: 模版不一致

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* feat: Support multi tools calling

* feat: Support multi tools calling

* fix: Add '\n' between each tool

* fix: Add ensure_ascii=False

* bugfix: rfind

* bugfix: tools_call -> tool_calls

* bugfix: add toolName in tool_response

* fix: some '\n' error

* fix: remove toolname

* fix: replace '\n' to self.separator

* feat: add doc with multiple tool calling

* fix：update doc

* feat: add qwen2.5 prompt template test

* feat: add qwen2.5 no tool call prompt test

---------

Co-authored-by: gaozixiang <[email protected]>

* Update supported models & Ascend doc (InternLM#2765)

* update ascend supported model list

* fix markdown

* fix markdown

* fix lint

* Update get_started.md

* Update get_started.md

* [CI] Split vl testcases into turbomind and pytorch backend (InternLM#2751)

* updaet

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* [Feature] support minicpm-v_2_6 for pytorch engine. (InternLM#2767)

* support minicpmv_2_6.

* update supported_models.

* update supported_models.

* Support qwen2-vl AWQ quantization (InternLM#2787)

* Support qwen2-vl AWQ quantization

* Update config.yaml

---------

Co-authored-by: zhulinJulia24 <[email protected]>

* [dlinfer] Fix qwenvl rope error for dlinfer backend (InternLM#2795)

* Optimize update_step_ctx on Ascend (InternLM#2804)

* opt update_ctx for ascend

* fix lint

---------

Co-authored-by: 逝夜长歌 <[email protected]>
Co-authored-by: gaozixiang <[email protected]>
Co-authored-by: jinminxi104 <[email protected]>
Co-authored-by: zhulinJulia24 <[email protected]>
Co-authored-by: zhoushenglong <[email protected]>
Co-authored-by: AllentDan <[email protected]>
Co-authored-by: Wei Tao <[email protected]>
@lvhan028
resolve conflicts
70875eb
@lvhan028
Merge pull request InternLM#2812 from lvhan028/resolve-conflicts
787f765 
Resolve conflicts
@grimoire
PytorchEngine refactor multimodal (InternLM#2742)
099721a 
* WIP

* support mrope

* support long context

* support causal=false

* fix mask

* flash attn bound

* optimize

* Moskau, Moskau, wirf die Gläser an die Wand

* YMCA

* optimize mllama

* update processor

* support cogvlm

* all work and no play make jack a dull boy

* upgrade triton

* support qwen2vl

* support internvl

* phi3-v WIP

* glm4v WIP

* support chatglm and cogvlm

* use image tokens

* support llava

* support internvl-mono

* phi3v, mllama

* add llavanext

* use img token ids

* support multiimage chatglm cogvlm

* fix ut
@lvhan028
minor-fix
36a15e3
@lvhan028
minor-fix (InternLM#2813)
4e6760e
@grimoire
fix
f7c167e
@lvhan028
fix
e977361
@grimoire
fix mono
5dc967d
@grimoire
fix docs
ae7015a
@grimoire
read norm_type
c746fd3
@lvhan028
Merge branch 'main' into refactor-vl
dc9757c
@lvhan028
super().collect_images->self.collect_images
d577acb
@lvhan028
Merge branch 'refactor-vl' of github.com:InternLM/lmdeploy into refac…
30ea075 
…tor-vl
@lvhan028
add note in supported models
bbcf9a5
@lvhan028
define the parameters clearly
b3a2887
@grimoire
better streaming
8f7a56f
@grimoire
merge main
c2a5b44
grimoire and others added 22 commits December 10, 2024 11:51
@grimoire
fix llava
92b09d0
@lvhan028
fix minicpm 2.6
db367f4
@grimoire
fix callback
4e2f1f8
@lvhan028
fix minicpm v2.5
715fbb3
@lvhan028
fix minicpm v2.6
1a6d88f
@lvhan028
Merge branch 'refactor-vl' into fix-refactor-vl
8ee759f
@lvhan028
update llava_next.py
f0a4422
@lvhan028
remove hardcode from xcomposer2.py
ee022ad
@lvhan028
Merge pull request InternLM#2879 from lvhan028/fix-refactor-vl
02a25eb 
Fix VLM batch inference error
@lvhan028
rollback supported_models
a21abe3
@lvhan028
change to staticmethod
6a9342e
@grimoire
optimize tp
798298b
@grimoire
solve conflict
1b6ea24
@grimoire
Merge branch 'refactor-vl' into optimize-tp-broadcast
a9aacda
@grimoire
Merge branch 'refactor-vl' of github.com:InternLM/lmdeploy into refac…
d005bc8 
…tor-vl
@grimoire
Merge branch 'refactor-vl' into optimize-tp-broadcast
e9517e1
@lvhan028
fix vlm quantization
4107d4f
@lvhan028
Merge branch 'refactor-vl' of github.com:InternLM/lmdeploy into refac…
fdaa601 
…tor-vl
@lvhan028
update doc
e5a5085
@lvhan028
update
bc93e73
@grimoire
Merge branch 'refactor-vl' into optimize-tp-broadcast
67188eb
@grimoire
solve conflict
6cdc76a
AllentDan
AllentDan approved these changes Dec 17, 2024
View reviewed changes
Copy link
Collaborator

@AllentDan AllentDan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested OK with Qwen2-VL

RunningLeon
RunningLeon approved these changes Dec 17, 2024
View reviewed changes
Copy link
Collaborator

@RunningLeon RunningLeon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lvhan028 lvhan028 merged commit 8afb84c into InternLM:main Dec 17, 2024
5 checks passed
@lvhan028
Copy link
Collaborator

Will it affect backends supported by dlinfer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AllentDan AllentDan AllentDan approved these changes

@RunningLeon RunningLeon RunningLeon approved these changes

Assignees
No one assigned
Labels
improvement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@grimoire @lvhan028 @RunningLeon @AllentDan