feat: Add Spark from_json function

[zhli1142015]

Why I Need to Reimplement JSON Parsing Logic Instead of Using CAST(JSON):

• Failure Handling:
  On failure, from_json(JSON) returns NULL. For instance, parsing {"a 1} would
  result in {NULL}.
• Type Restrictions:
  Only ROW, ARRAY, and MAP types are allowed as root types.
• Boolean Handling:
  Only true and false are considered valid boolean values. Numeric values or
  strings will result in NULL.
• Integral Type Handling:
  Only integral values are valid for integral types. Floating-point values and
  strings will produce NULL.
• Float/Double Handling:
  All numeric values are valid for float/double types. However, for strings, only
  specific values like "NaN" or "INF" are valid.
• Array Handling:
  Spark allows a JSON object as input for an array schema only if the array is
  the root type and its child type is a ROW.
• Map Handling:
  Keys in a MAP can only be of VARCHAR type. For example, parsing {"3": 3}
  results in {"3": 3} instead of {3: 3}.
• Row Handling:
  Spark supports partial output mode. However, it does not allow an input JSON
  array when parsing a ROW.

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	b1ad181
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/67b99dbb321dd00008d98822

[zhli1142015]

cc @rui-mo and @PHILO-HE , thanks.

[rui-mo]

Thanks. Added some initial comments.

[jinchengchenghh]

The json input is variable, how can we make sure all the implement matches to Spark, Maybe we need to search from_json in Spark and make sure the result is correct.

[zhli1142015]

The json input is variable, how can we make sure all the implement matches to Spark, Maybe we need to search from_json in Spark and make sure the result is correct.

The current implementation supports only Spark's default behavior, and we should fall back to Spark's implementation when specific unsupported cases arise. These include situations where user-provided options are non-empty, schemas contain unsupported types, schemas include a column with the same name as spark.sql.columnNameOfCorruptRecord, or the configuration spark.sql.json.enablePartialResults is disabled.

The only existing unit tests in Spark related to this function are found in JsonExpressionsSuite and JsonFunctionsSuite. I have verified that these tests pass and added missing tests to ensure the current implementation aligns with Spark's behavior. For further details, please refer to the new unit tests included in this PR.

[rui-mo]

Thanks.

[rui-mo]

Thanks for the update! Added some comments.

[rui-mo]

Thanks for iterating!

[PHILO-HE]

Looks basically good.
Are nested complex types supported? E.g., array element is an array, struct or map. It would be better to clarify this in document and add some tests if lacked. Thanks!

[zhli1142015]

Kindly ping for this PR, @PHILO-HE and @rui-mo , thanks.

[rui-mo]

Thanks! Added some nits.

[PHILO-HE]

Some trivial comments. Thanks!

[zhli1142015]

Thanks for the review, @rui-mo and @PHILO-HE. The CI failure is unrelated. Is it okay to merge?

[rui-mo]

Thanks!

[mbasmanova]

Is it the case that if JSON doesn't match the type specified in the "cast", the result is null? Would you document that?

Test name says "invalidJson" suggesting that it works with malformed JSON, but the JSON in the test seems perfectly valid.

[zhli1142015]

The test cases actually contain invalid JSON. Null is returned, because the input string is unparsable. The first case is missing the opening {, the second case has a key without double quotes, and the third case is missing a :. The valid JSON string is {"a": 1}.

[mbasmanova]

@zhli1142015 Got it. Thank you for clarifying. What happens if JSON is valid but doesn't match the type? E.g. JSON is an array, but type is a map or a struct?

[zhli1142015]

For this case, null is returned, test case structEmptyArray covers this.

[facebook-github-bot]

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@mbasmanova merged this pull request in 48e2ed7.