Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve most skipped unittests #559

Merged
merged 30 commits into from
Jan 22, 2025
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
20b1ffe
* fix missing attribute error and duplicate test funcs
HYLcool Dec 24, 2024
c57ca12
* fix unexpected keyword argument error
HYLcool Dec 24, 2024
38bfd06
* review for OOM tests
HYLcool Dec 25, 2024
8fb34b3
* review for OOM tests
HYLcool Dec 25, 2024
f63a80e
- use 2 np instead of 4 np for unittest
HYLcool Dec 25, 2024
8a4e822
- use 2 np instead of 4 np for unittest to resolve OOM problem
HYLcool Dec 25, 2024
13460e5
* update doc of analyzer
HYLcool Dec 26, 2024
4110a1a
* fix undefined device_map: using balanced in default or use `to` met…
HYLcool Dec 27, 2024
829a0c7
* fix bugs in video_captioning_from_summarizer_mapper due to reorgani…
HYLcool Dec 27, 2024
3aee531
- remove unused imports
HYLcool Dec 27, 2024
4805928
* open unittest for nlpcda_en_mapper
HYLcool Dec 30, 2024
6020a41
* set the default encoding of stdout to utf-8
HYLcool Dec 30, 2024
d464b79
* set the default encoding of stdout to utf-8
HYLcool Dec 30, 2024
8981b63
* set the default encoding of stdout to utf-8
HYLcool Dec 30, 2024
2025a2e
* test for raft
HYLcool Dec 30, 2024
85e757d
* change the thresholds
HYLcool Dec 30, 2024
b41833e
+ add mem_required for generate_qa_from_text_mapper
HYLcool Jan 9, 2025
fb503db
+ add mem_required for two ops
HYLcool Jan 13, 2025
1aa9d24
Merge branch 'main' into resolve/unittest_skipping
HYLcool Jan 14, 2025
432de0f
- open unittest for generate_qa_from_text_mapper
HYLcool Jan 15, 2025
06267ba
- open unittest for generate_qa_from_examples_mapper
HYLcool Jan 15, 2025
c56a629
Merge branch 'main' into resolve/unittest_skipping
HYLcool Jan 15, 2025
684666e
Merge branch 'main' into resolve/unittest_skipping
HYLcool Jan 20, 2025
bece0d1
* fix skip_op_error & update_sampling_params
HYLcool Jan 20, 2025
88e9aa4
* update vllm version requirement for generation_config param
HYLcool Jan 20, 2025
26d0f84
* skip vllm ops
HYLcool Jan 20, 2025
6874191
* open unittests for api-related ops
HYLcool Jan 21, 2025
42e54bc
Merge branch 'main' into resolve/unittest_skipping
HYLcool Jan 21, 2025
f52106d
* fix wrong attr name
HYLcool Jan 21, 2025
d2b0064
* increase shm_size to avoid OOM
HYLcool Jan 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
* change the thresholds
HYLcool committed Dec 30, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 85e757dad5f0208681bc83f226bff0d4bae44aee
15 changes: 5 additions & 10 deletions tests/ops/filter/test_video_motion_score_raft_filter.py
Original file line number Diff line number Diff line change
@@ -6,11 +6,7 @@
from data_juicer.ops.filter.video_motion_score_raft_filter import \
VideoMotionScoreRaftFilter
from data_juicer.utils.constant import Fields
from data_juicer.utils.unittest_utils import DataJuicerTestCaseBase, SKIPPED_TESTS

# skip due to conflicts when run lazy_load in multiprocessing in librosa
# tests passed locally.
# @SKIPPED_TESTS.register_module()
from data_juicer.utils.unittest_utils import DataJuicerTestCaseBase
class VideoMotionScoreRaftFilterTest(DataJuicerTestCaseBase):

data_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), '..',
@@ -26,7 +22,6 @@ def _run_helper(self, op, source_list, target_list, np=1):
column=[{}] * dataset.num_rows)
dataset = dataset.map(op.compute_stats, num_proc=np)
dataset = dataset.filter(op.process, num_proc=np)
print(dataset[Fields.stats])
dataset = dataset.select_columns(column_names=[op.video_key])
res_list = dataset.to_list()
self.assertEqual(res_list, target_list)
@@ -134,7 +129,7 @@ def test_middle(self):
'videos': [self.vid3_path]
}]
tgt_list = [{'videos': [self.vid2_path]}]
op = VideoMotionScoreRaftFilter(min_score=3, max_score=10.5)
op = VideoMotionScoreRaftFilter(min_score=3, max_score=10.2)
self._run_helper(op, ds_list, tgt_list)

def test_any(self):
@@ -151,7 +146,7 @@ def test_any(self):
'videos': [self.vid2_path, self.vid3_path]
}]
op = VideoMotionScoreRaftFilter(min_score=3,
max_score=10.5,
max_score=10.2,
any_or_all='any')
self._run_helper(op, ds_list, tgt_list)

@@ -165,7 +160,7 @@ def test_all(self):
}]
tgt_list = []
op = VideoMotionScoreRaftFilter(min_score=3,
max_score=10.5,
max_score=10.2,
any_or_all='all')
self._run_helper(op, ds_list, tgt_list)

@@ -181,7 +176,7 @@ def test_parallel(self):
'videos': [self.vid3_path]
}]
tgt_list = [{'videos': [self.vid2_path]}]
op = VideoMotionScoreRaftFilter(min_score=3, max_score=10.5)
op = VideoMotionScoreRaftFilter(min_score=3, max_score=10.2)
self._run_helper(op, ds_list, tgt_list, np=2)