Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

f16::from_f64 with hardware intrinsics gives wrong (or at least different) result for corner cases #116

Open
coolreader18 opened this issue Sep 25, 2024 · 1 comment

Comments

@coolreader18
Copy link

coolreader18 commented Sep 25, 2024

This got caught by RustPython's regression tests that we port from CPython. If this is expected/acceptable behavior, that's fine, since from_f64_const works anyways.

Here's the failing test cases adapted into rust (the original is here; the first one is line 848 and the second line 890). For this, the from_f64_const result is what the python regression tests were expecting, and what worked in half 1.x (without use-intrinsics enabled, at least).

// from the round-to-even section of the test case
let x = 2.0f64.powi(-14) - 2.0f64.powi(-25) - 2.0f64.powi(-65); // 0x3f0ffbfffffffffc (0.0006100535392761228)
f16::from_f64_const(x); // 0x03ff (0.00006097555)
f16::from_f64(x); // 0x0400 (0.000061035156)

// from the double rounding section of the test case
// comment from the cpython test case: should be 2047, if double-rounded 64>32>16, becomes 2048
let x = 0x1ffdffffffu64 as f64 * 2f64.powi(-26); // 0x409ffdffffff0000 (2047.4999999850988)
f16::from_f64_const(x); // 0x67ff (2047)
f16::from_f64(x); // 0x6800 (2048)
@Alexhuszagh
Copy link

Alexhuszagh commented Dec 13, 2024

This is because it uses _mm_cvtps_ph under the hood, which converts a f32 to a 16-bit size, so the f64 is first cast to f32 and then uses the hardware instructions. The f64 analog, _mm_cvtpd_ph, unfortunately, isn't stable (and of course require different hardware support).

Alexhuszagh added a commit to Alexhuszagh/float16 that referenced this issue Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants