Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve most skipped unittests #559

Merged
merged 30 commits into from
Jan 22, 2025
Merged

Resolve most skipped unittests #559

merged 30 commits into from
Jan 22, 2025

Conversation

HYLcool
Copy link
Collaborator

@HYLcool HYLcool commented Jan 22, 2025

  • unit test opt:
    • open most skipped unit tests
      • 20 unit tests for API-based OPs are all opened again by setting the API info with environment variables
      • 1 unit test for video_motion_score_raft_filter is opened again by setting more appropriate thresholds to handle different model outputs on different hardware, which is normal.
      • 7 unit tests for model-based OPs are opened again by decreasing the number of processors to avoid OOM of GPU memory.
    • increase the shm_size to 128G to avoid implicit OOM issue with exit code 137 (ref)
    • for now, there are still 9 unit tests that are skipped left to be resolved in future work
  • others:
    • add default mem_required for some model-based OPs
    • update sampling_params for vllm-based OPs due to vllm and transformers having similar inference parameters with different names.
    • update device_map specification for latest diffusers library.
    • fix some bugs & typos

…hod to move models to specified devices

* fix unrecognized dtype: only need torch.dtype instead of strings like 'fp16'
* open unittest for image_diffusion_mapper
# Conflicts:
#	configs/config_all.yaml
#	data_juicer/ops/aggregator/entity_attribute_aggregator.py
#	data_juicer/ops/aggregator/most_relavant_entities_aggregator.py
#	data_juicer/ops/aggregator/nested_aggregator.py
#	data_juicer/ops/mapper/video_captioning_from_summarizer_mapper.py
#	tests/ops/mapper/test_image_tagging_mapper.py
@HYLcool HYLcool added bug Something isn't working enhancement New feature or request dj:ci/cd issues/PRs about CI/CD of Data-Juicer environment related to third-party dependency, DJ-pypi, DJ-docker, etc. labels Jan 22, 2025
@HYLcool HYLcool self-assigned this Jan 22, 2025
Copy link
Collaborator

@yxdyc yxdyc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HYLcool HYLcool merged commit dbf880c into main Jan 22, 2025
4 checks passed
@HYLcool HYLcool deleted the resolve/unittest_skipping branch January 22, 2025 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dj:ci/cd issues/PRs about CI/CD of Data-Juicer enhancement New feature or request environment related to third-party dependency, DJ-pypi, DJ-docker, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants