Mixing int32 and int64 types in comparisons leads to errors (certainl… #1
Conversation
…y on x86_64 builds)
|
Thank you for this PR! Would you please also add a test that reproduces the issue on your system? Since you are using a specific bit configuration, I think a new file with just your test(s) would probably be best. Thanks! |
|
Simple test added. Should also point out that I'm compiling & testing on an x86_64 system. Same problem exists for arm64 builds - Don't have a native 32bit system to hand for testing. |
|
Thank you! Looks reasonable to me - I'll test on my systems and then merge. I have the same platforms available as you do, but it occurs to me I could probably cross-compile for i686 and try that as well. |
…values of 8, 64, ... The issue is down in part, to the return value of fixedpt_div being truncated in the conversion from fixedptd to fixedpt
|
Sorry for the delay --- I haven't forgotten; I've just been very busy at work! |
…y on x86_64 builds)
Built a small test routine calling fixedpt_sqrt(x) where 0>x<1 with -DFIXEDPT_BITS=64 -DFIXEDPT_WBITS=14
Returned a pathological and incorrect result:
fixedpt_sqrt(0.070700)) = 7314.706333859 (expected 0.265895)
Comparison & conversions of int32 & int64 values to blame .