Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Neva ETP EPP support #12154

Open
wants to merge 103 commits into
base: main
Choose a base branch
from
Open

Neva ETP EPP support #12154

wants to merge 103 commits into from

Conversation

yaoyu-33
Copy link
Collaborator

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

yaoyu-33 and others added 30 commits December 12, 2024 11:20
@yaoyu-33
api updates and fixes
15439bf 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
6bfd873 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
fix
773b4c9 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
fix arg
3a1a017 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
update seq packing in mock ds
e3e87b7 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge branch 'main' into yuya/neva2_seq_packing
4ee633c
@yaoyu-33
Merge branch 'main' into yuya/neva2_seq_packing
ecc813d
@yaoyu-33
save
c10157c 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
update preprocess_data
84eb7cc 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
update seq packing
3bf6442 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
c8a26af 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
fix sp
48b5261 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge remote-tracking branch 'origin/yuya/neva2_seq_packing' into yuy…
365c051 
…a/neva2_seq_packing
@yaoyu-33
save
7da82ed 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge branch 'main' into yuya/neva2_seq_packing
4127c40
@yaoyu-33
fix seq packing
c5d26c3 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
add truncation and padding
ecd461f 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
5e0a168 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Fix issues
9240a79 
Signed-off-by: yaoyu-33 <[email protected]>
@yashaswikarnati
change LLaVATemplateConfig variables to class variables
4808999
@yashaswikarnati
change to use field with default attributes
c4d92f9
@yashaswikarnati
Apply isort and black reformatting
ad44132 
Signed-off-by: yashaswikarnati <[email protected]>
@yaoyu-33
Merge remote-tracking branch 'origin/yuya/neva2_seq_packing' into yuy…
7db8e52 
…a/neva2_seq_packing

# Conflicts:
#	nemo/collections/vlm/neva/model/base.py
@yaoyu-33
Apply isort and black reformatting
e705afe 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge remote-tracking branch 'origin/yash/fix_template_dataclass' int…
f0a9cb5 
…o yuya/neva2_seq_packing
@parthmannan
Initial support for CP
f508f8b
@yaoyu-33
Add seq packing option in energon
7415036 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Fix energon conversation
af1f32a 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
add energon option in neva training script
568f9aa 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
01fd6cf 
Signed-off-by: yaoyu-33 <[email protected]>
yaoyu-33 and others added 27 commits January 15, 2025 13:44
@yaoyu-33
Merge remote-tracking branch 'origin/main' into pmannan/neva_cp
ae6f5f7 
# Conflicts:
#	nemo/collections/vlm/neva/model/base.py
@yaoyu-33
revert overlap config change
65233cb 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
fix neva generate
efbbdbf 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
remove not used module
5517510 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
update encoder / decoder seq len settings
49854a0 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
temp save
5d094a6 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge branch 'main' into yuya/refactor_vlm_vision_module
bc52537 
# Conflicts:
#	nemo/collections/vlm/neva/model/base.py
@yaoyu-33
update logging
309461f 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
a610a88 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
update init / fix unused layer bug
e4fe166 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
c33bf50 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
remove not used import
edd6aab 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge remote-tracking branch 'origin/yuya/refactor_vlm_vision_module'…
4fad475 
… into yuya/refactor_vlm_vision_module
@yaoyu-33
Update for Siglip
1c47a2d 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
25f1247 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
update init
e7600e3 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge remote-tracking branch 'origin/yuya/refactor_vlm_vision_module'…
01e4cd7 
… into yuya/refactor_vlm_vision_module

# Conflicts:
#	nemo/collections/vlm/vision/__init__.py
#	nemo/collections/vlm/vision/test_import_ckpt.py
@yaoyu-33
Apply isort and black reformatting
46b2332 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Fix logging
c364196 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Apply isort and black reformatting
cfff3e0 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge branch 'yuya/refactor_vlm_vision_module' into yuya/neva_epp_etp
8e9d8ed 
# Conflicts:
#	scripts/vlm/neva_finetune.py
@yaoyu-33
fix hang
789dfe8 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
fix
e5d85f2 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
fix hang again
4932205 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
fix few issues
cfe337f 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
turn off overlap
3296be1 
Signed-off-by: yaoyu-33 <[email protected]>
@yaoyu-33
Merge branch 'refs/heads/main' into yuya/neva_epp_etp
462fd23 
# Conflicts:
#	nemo/collections/vlm/__init__.py
#	nemo/collections/vlm/neva/model/base.py
#	nemo/collections/vlm/vision/base.py
#	nemo/collections/vlm/vision/clip_vit.py
#	nemo/collections/vlm/vision/intern_vit.py
#	nemo/collections/vlm/vision/siglip_vit.py
#	scripts/vlm/neva_finetune.py
github-advanced-security[bot]
github-advanced-security bot found potential problems Feb 12, 2025
View reviewed changes
scripts/vlm/neva_finetune.py
@@ -43,6 +43,7 @@
from nemo.collections import llm, vlm
from nemo.collections.multimodal.data.energon.task_encoder import MultiModalTaskEncoder
from nemo.collections.vlm import ImageDataConfig
from nemo.lightning.pytorch.callbacks import NsysCallback

Check notice

Code scanning / CodeQL

Unused import Note

Import of 'NsysCallback' is not used.

Copilot Autofix AI 2 days ago

To fix the problem, we need to remove the unused import statement. This will clean up the code and eliminate the unnecessary dependency. Specifically, we will delete the line that imports NsysCallback from nemo.lightning.pytorch.callbacks.

Suggested changeset 1
scripts/vlm/neva_finetune.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/scripts/vlm/neva_finetune.py b/scripts/vlm/neva_finetune.py
--- a/scripts/vlm/neva_finetune.py
+++ b/scripts/vlm/neva_finetune.py
@@ -45,3 +45,3 @@
 from nemo.collections.vlm import ImageDataConfig
-from nemo.lightning.pytorch.callbacks import NsysCallback
+
 from nemo.lightning.pytorch.callbacks.megatron_comm_overlap import MegatronCommOverlapCallback
EOF
@@ -45,3 +45,3 @@
from nemo.collections.vlm import ImageDataConfig
from nemo.lightning.pytorch.callbacks import NsysCallback

from nemo.lightning.pytorch.callbacks.megatron_comm_overlap import MegatronCommOverlapCallback
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Multi Modal
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yaoyu-33 @yashaswikarnati @parthmannan