You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Both IdentityToZeroTransformation and NullToZeroTransformation are to handle special instances where LinearTransformer is used to map Numeric columns, but the values are either identical or all null. Ideally, these should be superseded when appending "regular" data by LinearTransformation instances. For now, it is not the case.
How to reproduce?
For IdentityToZeroTransformation for instance(and similarly for NullToZeroTransformation):
IdentityTransformation should NOT be superseded by another IdentityTransformation. (By definition, the space value of Identity A is not considered in Identity B unless value a and value b are the same).
IdentityTransformation should NOT superseded by a NullToZeroTransformation. Same case as the Identity.
IdentityTransformation might be superseded by a LinearTransformation if max and min cover the identity value.
Now, in cases 1 and 2, we might require a trigger of another type of transformation, such as LinearTransformation, in which we include values from A and B as the ranges.
What went wrong?
Both
IdentityToZeroTransformation
andNullToZeroTransformation
are to handle special instances whereLinearTransformer
is used to mapNumeric
columns, but the values are either identical or allnull
. Ideally, these should be superseded when appending "regular" data byLinearTransformation
instances. For now, it is not the case.How to reproduce?
For
IdentityToZeroTransformation
for instance(and similarly forNullToZeroTransformation
):2. Branch and commit id:
main
, f066acf3. Spark version:
3.4.1
4. Hadoop version:
3.3.4
5. How are you running Spark?
Locally
The text was updated successfully, but these errors were encountered: