GH-31869: [Python][Parquet] First pass at implementing external key material features in python

[pfparsons]

Rationale for this change

Enables external key material and rotation for individual parquet files in PyArrow. This change does not address any parquet dataset encryption functionality. (though I would be happy to keep going with any work required for other applications of external store including the dataset API once I get some feedback on this PR)

What changes are included in this PR?

This PR enables external key material for parquet encryption from PyArrow:
Optional parquet_file_path and FileSystem paramters to CryptoFactory - mirroring the interface for CryptoFactory in C++

1. Exposes the rotate_master_keys method of CryptoFactory

2. Adds Cython classes for FileKeyMaterialStore, FileSystemKeyMaterialStore, and KeyMaterial - but does not expose these from PyArrow encryption. I included these changes only so that a unit test may verify an external store without leaking the implementation details for the store into the test.

Are these changes tested?

Yes - I've modified an existing test (previously marked pytest.xfail) to do a basic read write test and verify creation of the external key material store.

I would like to get some feedback as to whether this change is welcome and headed in the right direction before going any further.

Are there any user-facing changes?

This change enables users to optionally supply a parquet file path and FileSystem to CryptoFactory methods that provide en/decryption_properties.

• GitHub Issue: [Python][Parquet] Enable usage of external key material and rotation for encryption keys in PyArrow #31869

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

[pfparsons]

Hi @raulcd would you mind taking a look at this?

[raulcd]

Thanks for the PR @pfparsons !

I am not too familiar with the Parquet encryption side of things but I'll try to take a look. I am running CI at the moment. Maybe @EnricoMi @rok @adamreeve who have worked on Parquet Encryption in the past can take a look

[adamreeve]

Hi @pfparsons, thanks for the contribution! There are some minor formatting issues that the pre-commit script is reporting in https://github.com/apache/arrow/actions/runs/19077444221/job/54553464743?pr=48009, but the other CI failures are unrelated and are failing on main at the moment.

You can install pre-commit and run it locally to reproduce these errors, see https://pre-commit.com/. You can use something like pre-commit run --from-ref main --to-ref feature/31869 to run it on changes already committed.

I've left some comments with suggested changes, but this mostly looks really good to me. I'm not super familiar with cython though.

[adamreeve]

Could you also please add a test of key rotation?