GH-36412: [Python][CI] Fix extra deprecation warnings in the pandas nightly build

[AlenkaF]

Fixes left deprecation warnings coming from the pandas development version, by updating our test code to avoid the deprecated patterns.

• Closes: [Python][CI] Fix deprecation warnings in the pandas nightly build #36412

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10-pandas-nightly

[github-actions]

Revision: 7230b6f

Submitted crossbow builds: ursacomputing/crossbow @ actions-40718f16dc

Task	Status
test-conda-python-3.10-pandas-nightly

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10-pandas-nightly

[github-actions]

Revision: 0d7edf3

Submitted crossbow builds: ursacomputing/crossbow @ actions-f2bde2fa12

Task	Status
test-conda-python-3.10-pandas-nightly

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10-pandas-nightly

[github-actions]

Revision: 09f2f50

Submitted crossbow builds: ursacomputing/crossbow @ actions-3a8abeda7e

Task	Status
test-conda-python-3.10-pandas-nightly

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10-pandas-nightly

[github-actions]

Revision: 3adf04e

Submitted crossbow builds: ursacomputing/crossbow @ actions-5666ccb6e4

Task	Status
test-conda-python-3.10-pandas-nightly

[AlenkaF]

@github-actions crossbow submit test-conda-python-3.10-pandas-nightly

[github-actions]

Revision: 01980c1

Submitted crossbow builds: ursacomputing/crossbow @ actions-db434e524e

Task	Status
test-conda-python-3.10-pandas-nightly

[AlenkaF]

I think this PR looks good now.

The only missing thing I have to solve is this warning about comparison with empty arrays (coming from pandas.testing.assert_frame_equal):

=============================== warnings summary ===============================
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_pandas.py::TestConvertMetadata::test_empty_list_metadata
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_pandas.py::TestConvertListTypes::test_empty_list_roundtrip
  /opt/conda/envs/arrow/lib/python3.10/site-packages/pandas/core/dtypes/missing.py:578: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
    return lib.array_equivalent_object(left, right)

opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_pandas.py: 2 warnings
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/parquet/test_data_types.py: 3 warnings
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/parquet/test_pandas.py: 2 warnings
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/parquet/test_parquet_file.py: 6 warnings
  /opt/conda/envs/arrow/lib/python3.10/site-packages/pandas/core/dtypes/missing.py:579: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
    if not lib.array_equivalent_object(left[~mask], right[~mask]):

But I would do this in a separate PR.

[jorisvandenbossche]

Thanks!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 6eeee3b.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.