-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Tests] Fix tests 3rd ed. #215
Conversation
ebcbc42
to
a677d07
Compare
src/flag_gems/testing/__init__.py
Outdated
def assert_close(res, ref, dtype, equal_nan=False, reduce_dim=1): | ||
ref = ref.to(dtype) | ||
atol = 1e-4 * reduce_dim | ||
rtol = RESOLUTION[dtype] | ||
torch.testing.assert_close(res, ref, atol=atol, rtol=rtol, equal_nan=equal_nan) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two issues:
- fp64 is not in RESOLUTION's key set.
- passing a dtype matching neither ref nor res may raise an error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, do we consider default dtype=None? Now dtype is mandate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two issues:
- fp64 is not in RESOLUTION's key set.
- passing a dtype matching neither ref nor res may raise an error.
Now only isclose & allclose test fp64. These two tests do not use assert_close.
Maybe we can assert res.dtype == dtype here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two issues:
- fp64 is not in RESOLUTION's key set.
- passing a dtype matching neither ref nor res may raise an error.
Now only isclose & allclose test fp64. These two tests do not use assert_close. Maybe we can assert res.dtype == dtype here?
done
tests/test_binary_pointwise_ops.py
Outdated
@@ -21,6 +21,11 @@ | |||
from .conftest import TO_CPU | |||
|
|||
|
|||
def remove_zero(inp): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would rename this as replace_zeros
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would rename this as replace_zeros
done
if upcast: | ||
ref_inp = ref_inp.to(torch.float64) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about integer? we are not supposed to upcast integers to floats in all occasions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about integer? we are not supposed to upcast integers to floats in all occasions.
Now to_reference is only for float inputs.
if TO_CPU: | ||
a = a.to("cpu") | ||
assert torch.equal(a, b) | ||
def gems_assert_close(res, ref, dtype, equal_nan=False, reduce_dim=1): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what's different between this and testing.assert_close?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what's different between this and testing.assert_close?
TO_CPU here.
tools/op-unit-test.sh
Outdated
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_special_ops.py --ref=cpu --mode=quick && \ | ||
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_pointwise_type_promotion.py --ref=cpu --mode=quick && \ | ||
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_tensor_constructor_ops.py --ref=cpu --mode=quick && \ | ||
bash tools/pytest_mark_check.sh &" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's better to put pytest_mark_check.sh
before running the unit tests. Or use an additional ci pipeline named pytest_mark_check
to run this check
tools/op-unit-test.sh
Outdated
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_binary_pointwise_ops.py --ref=cpu --mode=quick && \ | ||
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_special_ops.py --ref=cpu --mode=quick && \ | ||
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_pointwise_type_promotion.py --ref=cpu --mode=quick && \ | ||
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_tensor_constructor_ops.py --ref=cpu --mode=quick && \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_tensor_constructor_ops.py --ref=cpu --mode=quick && \ | |
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_tensor_constructor_ops.py --ref=cpu --mode=quick & |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add to CONTRIBUTING.md
3ea8f84
to
e2afd8c
Compare
tools/op-unit-test.sh
Outdated
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_special_ops.py --ref=cpu --mode=quick && \ | ||
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_pointwise_type_promotion.py --ref=cpu --mode=quick && \ | ||
CUDA_VISIBLE_DEVICES=7 coverage run ${COVERAGE_ARGS} -m pytest -s tests/test_tensor_constructor_ops.py --ref=cpu --mode=quick &" | ||
"bash tools/pytest_mark_check.sh &" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[pytest]
filterwarnings =
ignore::pytest.PytestUnknownMarkWarning
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[pytest] filterwarnings = ignore::pytest.PytestUnknownMarkWarning
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
1. Change --device=cpu to --ref=cpu 2. Add --mode=quick to test only one shape for CI cpu tests. 3. Add scalar shape for pointwise_dynamic op tests 4. Move assert_close and assert_equal to flag_gems.testing 5. Add mark for each op test, and register it to pytest.ini 6. Add speedup in benchmark print result. --------- Co-authored-by: zhengyang <[email protected]>
PR Category
OP Test
Type of Change
Tests
Description
Issue
Progress
Performance