Fix panic on lossy decimal to float casting: round to saturation for overflows #7887

kosiew · 2025-07-09T09:00:54Z

Which issue does this PR close?

Closes #7886.

Rationale for this change

Casting large Decimal256 values to Float64 can exceed the representable range of floating point numbers. Previously, this could result in a panic due to unwrapping a failed conversion.

This PR introduces a safe conversion that saturates overflowing values to INFINITY or -INFINITY, following standard floating point semantics. This ensures stable, predictable behavior without runtime crashes.

What changes are included in this PR?

Introduced a helper function decimal256_to_f64 that converts i256 to f64, returning INFINITY or -INFINITY when the value is out of range.
Updated the casting logic for Decimal256 → Float64 to use the new safe conversion.
Improved inline and module-level documentation to reflect that this conversion is lossy and saturating.
Added a unit test test_cast_decimal256_to_f64_overflow to validate overflow behavior.

Are there any user-facing changes?

Yes.

Behavior Change: When casting Decimal256 values that exceed the f64 range, users now receive INFINITY or -INFINITY instead of a panic.
Improved Docs: Updated documentation clarifies the lossy and saturating behavior of decimal-to-float casting.
Not a Breaking Change: There are no API changes, but users relying on panics for overflow detection may observe different behavior.

…ures

… conversion process and error handling

… detailed context in error messages, such as failing element index and input value.

tustvold · 2025-07-09T09:13:33Z

Is it possible that the issue is that the float conversion is fallible, I wouldn't have expected the conversion to be fallible? Can we fix this? Changing to try_unary will significantly regress performance and may be the wrong fix?

… include detailed context in error messages, such as failing element index and input value." This reverts commit cf9268d.

…d Float64 conversions

…6 to Float32

kosiew · 2025-07-09T09:30:33Z

Thanks @tustvold for the quick feedback

✅ What we’re doing now (safe, slow):

Replace .unwrap() with proper error handling.
Use try_unary to safely propagate conversion errors.
This is correct, but may slow things down.

🛠 Alternative approaches (fast, risky or complex):

Clamp or saturate
- Convert what we can, and clamp values to f64::MAX or f64::MIN when overflow is detected.
- Might hide data corruption; unsafe for financial data.
Ignore errors silently
- Keep using .unary() and fallback to 0.0 or NaN on error.
- Fast but potentially dangerous and violates correctness expectations.
Make overflow handling configurable
- Introduce a CastOptions flag like allow_float_overflow.
- Use unary() when allowed, try_unary() when strict.
Which of the above (or another option) would you recommend?

tustvold · 2025-07-09T09:34:08Z

I would expect us to follow standard floating point overflow behaviour. Ultimately if you're opting for floating point numbers you are opting for this behaviour. Floating point in general is not appropriate for financial data as it is lossy by design.

…ion failures" This reverts commit f25ec6b3ba8561f8a66b276d9d7869f8636ce48c.

…f error on overflow - Changed decimal-to-float casts to use lossy conversion consistent with IEEE semantics, saturating to ±INFINITY instead of returning an error on overflow or out-of-range values. - Updated `cast_decimal_to_float` to use infallible conversion function signature. - Added `decimal256_to_f64` helper for Decimal256 to f64 conversion with saturation. - Adjusted casting logic in `cast_with_options` accordingly. - Removed tests that expected errors on decimal-to-float overflow since now conversion saturates. - Clarified documentation to specify that decimal to float casts are lossy and saturate on overflow.

klion26 · 2025-07-10T04:56:33Z

arrow-cast/src/cast/mod.rs

@@ -8660,6 +8673,16 @@ mod tests {
            "did not find expected error '{expected_error}' in actual error '{err}'"
        );
    }
+    #[test]
+    fn test_cast_decimal256_to_f64_overflow() {


Does this need to cover negative infinity?

@klion26

Good catch.
I amended the test.

Could you also please either add or ensure there is an existing test for casting Decimal128 (i128::MIN and i128::MAX to f64)

… casting Decimal256 to Float64

alamb

Thanks @kosiew @tustvold and @klion26 -- this looks like a clear improvement to me (no panics 👏 )

I think we should add the equivalent test for Decimal128 but I don't think it is required for this PR (we could do it in another PR)

alamb · 2025-07-10T15:54:07Z

arrow-cast/src/cast/mod.rs

@@ -891,7 +893,7 @@ pub fn cast_with_options(
                scale,
                from_type,
                to_type,
-                |x: i256| x.to_f64().unwrap(),
+                |x: i256| decimal256_to_f64(x),


Do we need to do something similar for Decimal128 above?

alamb · 2025-07-10T15:54:48Z

arrow-cast/src/cast/mod.rs

@@ -1993,6 +1995,17 @@ where
    }
 }

+/// Convert a [`i256`] to `f64` saturating to infinity on overflow.
+fn decimal256_to_f64(v: i256) -> f64 {


Saturating to INF seems a better solution than panic'ing

alamb · 2025-07-10T15:55:55Z

arrow-cast/src/cast/mod.rs

@@ -8660,6 +8673,16 @@ mod tests {
            "did not find expected error '{expected_error}' in actual error '{err}'"
        );
    }
+    #[test]
+    fn test_cast_decimal256_to_f64_overflow() {


Could you also please either add or ensure there is an existing test for casting Decimal128 (i128::MIN and i128::MAX to f64)

Enhance decimal casting functions to return errors on conversion fail…

cb41f13

…ures

github-actions bot added the arrow Changes to the arrow crate label Jul 9, 2025

kosiew added 2 commits July 9, 2025 17:05

Enhance documentation for cast_decimal_to_float function to clarify…

89d360e

… conversion process and error handling

Enhance error handling in cast_decimal_to_float function to include…

cf9268d

… detailed context in error messages, such as failing element index and input value.

kosiew added 4 commits July 9, 2025 17:13

Revert "Enhance error handling in cast_decimal_to_float function to…

8ca4168

… include detailed context in error messages, such as failing element index and input value." This reverts commit cf9268d.

made the code uniform by using the .map() pattern for both Float32 an…

1bbfae3

…d Float64 conversions

Add test for casting Decimal128 to Float64 with overflow handling

c658501

Add tests for overflow handling when casting Decimal128 and Decimal25…

8e74ff2

…6 to Float32

kosiew added 2 commits July 9, 2025 19:27

Revert "Enhance decimal casting functions to return errors on convers…

3217813

…ion failures" This reverts commit f25ec6b3ba8561f8a66b276d9d7869f8636ce48c.

kosiew changed the title ~~Improve Decimal to Float Casting with Error Propagation and Overflow Handling~~ Add lossy decimal to float casting with saturation for overflows in Arrow Jul 9, 2025

kosiew marked this pull request as ready for review July 9, 2025 13:58

klion26 reviewed Jul 10, 2025

View reviewed changes

test(decimal cast): add tests for positive and negative overflow when…

05ace0c

… casting Decimal256 to Float64

alamb changed the title ~~Add lossy decimal to float casting with saturation for overflows in Arrow~~ Fix panic on lossy decimal to float casting: round to saturation for overflows Jul 10, 2025

alamb approved these changes Jul 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix panic on lossy decimal to float casting: round to saturation for overflows #7887

Fix panic on lossy decimal to float casting: round to saturation for overflows #7887

kosiew commented Jul 9, 2025 •

edited

Loading

Uh oh!

tustvold commented Jul 9, 2025

Uh oh!

kosiew commented Jul 9, 2025

Uh oh!

tustvold commented Jul 9, 2025

Uh oh!

klion26 Jul 10, 2025

Uh oh!

kosiew Jul 10, 2025

Uh oh!

alamb Jul 10, 2025

Uh oh!

alamb left a comment

Uh oh!

alamb Jul 10, 2025

Uh oh!

alamb Jul 10, 2025

Uh oh!

alamb Jul 10, 2025

Uh oh!

Uh oh!

Fix panic on lossy decimal to float casting: round to saturation for overflows #7887

Are you sure you want to change the base?

Fix panic on lossy decimal to float casting: round to saturation for overflows #7887

Conversation

kosiew commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

tustvold commented Jul 9, 2025

Uh oh!

kosiew commented Jul 9, 2025

✅ What we’re doing now (safe, slow):

🛠 Alternative approaches (fast, risky or complex):

Uh oh!

tustvold commented Jul 9, 2025

Uh oh!

klion26 Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

kosiew Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kosiew commented Jul 9, 2025 •

edited

Loading