Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the eval_use_gather_object flag usage #36214

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix the eval_use_gather_object flag usage #36214

wants to merge 1 commit into from

Conversation

ducha-aiki
Copy link
Contributor

What does this PR do?

Fixes #36213

After the first eval, the 2nd eval is crashing, while trying to concat batches with different shapes, as if the flag eval_use_gather_object stopped working. In evaluation_loop, the self.gather_function is reset, and the eval_use_gather_object is not used

This PR fixes that, using the same code line, which is used at accelerator creation.

@muellerzr and @SunMarc

@ducha-aiki
Copy link
Contributor Author

The test failure seems to be unrelated:

=================================== FAILURES ===================================
______________ LlavaForConditionalGenerationModelTest.test_config ______________
[gw3] linux -- Python 3.9.21 /usr/local/bin/python3

self = <tests.models.llava.test_modeling_llava.LlavaForConditionalGenerationModelTest testMethod=test_config>

    def test_config(self):
>       self.config_tester.run_common_tests()

tests/models/llava/test_modeling_llava.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_configuration_common.py:205: in run_common_tests
    self.check_config_can_be_init_without_params()
tests/test_configuration_common.py:169: in check_config_can_be_init_without_params
    config = self.config_class()
E   AssertionError: ValueError not raised

SunMarc
SunMarc approved these changes Feb 17, 2025
View reviewed changes
Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR ! LGTM ! Could you try to craft a test that fails before this PR ? Maybe you can take inspiration from test_eval_use_gather_object in test_trainer.py ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SunMarc SunMarc SunMarc approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[bug] use_gather_object is not respected after the first eval in trainer
2 participants
@ducha-aiki @SunMarc