You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
First, the code on line 12 is run if rounded is set to true. But clearly this line computes the output without rounding. It is only the else branch that does something containing rouding. I suspect the logic here is inverted.
Second, on line 14, where rounding is performed, the digits parameter is set to DIGITS. But it is unclear to me where this DIGITS comes from - it is not entered while calling the normalize function. Is it intended to be a global variable? EDIT: I see, it is a const set in the module to 20. It does not work anyway as I would anticipate. The code below:
julia> h =Decimal(0,2540,-4)
Decimal(0, 2540, -4)
julia> f =normalize(h)
Decimal(0, 254, -3)
julia> f =normalize(h,rounded=false)
Decimal(0, 254, -3)
julia> f =normalize(h,rounded=true)
Decimal(0, 254, -3)
julia> DIGITS=11
julia> f =normalize(h,rounded=true)
Decimal(0, 254, -3)
For comparision with a standard float:
julia>round(0.254,digits=1)
0.3
I am running this in REPL.
The text was updated successfully, but these errors were encountered:
I am checking the code for the
normalize
function at https://github.com/JuliaMath/Decimals.jl/blob/master/src/norm.jl and I suspect there are two problems in how therounded
option is handled.First, the code on line 12 is run if
rounded
is set totrue
. But clearly this line computes the output without rounding. It is only theelse
branch that does something containing rouding. I suspect the logic here is inverted.Second, on line 14, where rounding is performed, the
digits
parameter is set toDIGITS
. But it is unclear to me where thisDIGITS
comes from - it is not entered while calling thenormalize
function. Is it intended to be a global variable? EDIT: I see, it is aconst
set in the module to 20. It does not work anyway as I would anticipate. The code below:For comparision with a standard float:
I am running this in REPL.
The text was updated successfully, but these errors were encountered: