PARQUET-535: Make writeAllFields more efficient

[garlicbulbxian]

This will improve the write performance by 1/3 - 1/4 based on my testing. It makes a huge difference when dealing with really large files (several GBs).

In the original implementation, a significant time was spent on creating unnecessary Java objects.

[lw-lin]

Thanks for the patch @garlicccbulb ! I'll look into this and run some tests.

Could you add a Parquet JIRA issue for this and add it to this PR's summary?
We only merge PRs that start like "PARQUET-NNN: Make writeAllFields more efficient".

[garlicbulbxian]

Thanks @proflin., I created the jira in PARQUT queue, https://issues.apache.org/jira/browse/PARQUET-535.

Let me know if anything else to do.

[garlicbulb-puzhuo]

Can someone take a quick look at this?

[julienledem]

This looks good to me.
@lukasnalezenec: comments?

[julienledem]

@lw-lin @lukasnalezenec: any more comments?

[garlicbulb-puzhuo]

@julienledem @lw-lin @lukasnalezenec Sorry this one has been delayed for a while. I updated the PR to work with the new semantics with protobuf extensions. In a nutshell, there isn't much to do with extensions, due to the following facts.

• Whether there are extension fields for a given extendable protobuf schema can only be checked at runtime.
• The public API of protobuf provides to check the above needs the full deserialization and creations of the wrapper maps.

However, we can still benefit from the improvement from non-extendable protobuf schemas for large files (millions of rows).

[garlicbulb-puzhuo]

@julienledem @lw-lin @lukasnalezenec Any thoughts?