Skip to content

Commit

Permalink
[SPARK-37556][SQL] Deser void class fail with Java serialization
Browse files Browse the repository at this point in the history
**What changes were proposed in this pull request?**
Change the deserialization mapping for primitive type void.

**Why are the changes needed?**
The void primitive type in Scala should be classOf[Unit] not classOf[Void]. Spark erroneously [map it](https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala#L80) differently than all other primitive types. Here is the code:
```
private object JavaDeserializationStream {
  val primitiveMappings = Map[String, Class[_]](
    "boolean" -> classOf[Boolean],
    "byte" -> classOf[Byte],
    "char" -> classOf[Char],
    "short" -> classOf[Short],
    "int" -> classOf[Int],
    "long" -> classOf[Long],
    "float" -> classOf[Float],
    "double" -> classOf[Double],
    "void" -> classOf[Void]
  )
}
```
Spark code is Here is the demonstration:
```
scala> classOf[Long]
val res0: Class[Long] = long

scala> classOf[Double]
val res1: Class[Double] = double

scala> classOf[Byte]
val res2: Class[Byte] = byte

scala> classOf[Void]
val res3: Class[Void] = class java.lang.Void  <--- this is wrong

scala> classOf[Unit]
val res4: Class[Unit] = void <---- this is right
```

It will result in Spark deserialization error if the Spark code contains void primitive type:
`java.io.InvalidClassException: java.lang.Void; local class name incompatible with stream class name "void"`

**Does this PR introduce any user-facing change?**
no

**How was this patch tested?**
Changed test, also tested e2e with the code results deserialization error and it pass now.

Closes apache#34816 from daijyc/voidtype.

Authored-by: Daniel Dai <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
  • Loading branch information
daijyc authored and srowen committed Dec 7, 2021
1 parent f6e2e65 commit fb40c0e
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ private object JavaDeserializationStream {
"long" -> classOf[Long],
"float" -> classOf[Float],
"double" -> classOf[Double],
"void" -> classOf[Void])
"void" -> classOf[Unit])

}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,5 +69,5 @@ private class ContainsPrimitiveClass extends Serializable {
val floatClass = classOf[Float]
val booleanClass = classOf[Boolean]
val byteClass = classOf[Byte]
val voidClass = classOf[Void]
val voidClass = classOf[Unit]
}

0 comments on commit fb40c0e

Please sign in to comment.