GH-35243: [C#] Implement MapType

[CurtHagenlocher]

What changes are included in this PR?

This change includes the original work by @Platob from #35263, updated with the latest changes in the main branch. Additionally, it includes support in the C API for the Map type and fills a few gaps in functionality from the original change (for instance, ArrowArrayConcentrator support). Finally, it fixes a bug in the ArrowArrayConcentrator.Concatenate which prevented it from working correctly for structs. (This bug was discovered in the course of creating a test for concatenating maps.)

Are these changes tested?

Yes.

Are there any user-facing changes?

Maps are now supported in the C# implementation.

Closes #35243

• Closes: [C#] Implement MapType #35243

[davidhcoe]

a few nits on comments but looks good

[davidhcoe]

@lidavidm - are you able to help at all?

[lidavidm]

Thanks for the ping - I'll look at this tomorrow!

[lidavidm]

LGTM, thanks!

I restarted the integration pipeline so let's make sure things pass there.

[lidavidm]

Hmm, there appears to be something wrong with the pipeline

Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (setup.py): started
  Building wheel for pythonnet (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [59 lines of output]
      /tmp/pip-install-m_41mbf1/pythonnet_3828907f28a2453cac8f6ff2d84ed995/setup.py:468: SyntaxWarning: invalid escape sequence '\*'
        "MSBuild\**\Bin\MSBuild.exe",
      /tmp/pip-install-m_41mbf1/pythonnet_3828907f28a2453cac8f6ff2d84ed995/setup.py:533: SyntaxWarning: invalid escape sequence '\*'
        "MSBuild\**\Bin\MSBuild.exe",
      running bdist_wheel
      running build
      running build_ext
      /bin/sh: 1: mono: not found

[lidavidm]

Ah, weird, it tried to install 2.5.2 instead of 3.0.2.

@CurtHagenlocher can you rebase/merge one last time? It appears we recently had to pin Python to keep pythonnet working

[CurtHagenlocher]

Looks like another infrastructure problem

################# FAILURES #################
FAILED TEST: recursive_nested Rust producing,  Rust consuming
<class 'RuntimeError'>: Command failed: /arrow/rust/target/debug/flight-test-integration-client --host localhost --port=46587 --path /tmp/arrow-integration-o7e_thtg/generated_recursive_nested.json
With output:
--------------
Error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })))

[lidavidm]

Well, the C# tests pass so I think let's merge this.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 4b795ec.

There was 1 benchmark result indicating a performance regression:

• Commit Run on arm64-m6g-linux-compute at 2023-10-06 07:54:03Z
  • dataset-select (Python) with dataset=nyctaxi_multi_parquet_s3_repartitioned

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.