feat: Experimental credential_provider argument for scan_parquet

[nameexhaustion]

ref Allow Overriding Object Store Credential Provider #18979

Adds an experimental credential_provider argument to scan_parquet, which can accept a callable:

def credential_provider() -> tuple[dict[str, str | None], int | None]

Where the dictionary contains the credentials, and the optional 2nd value represents the expiry time as seconds since unix epoch, allowing us to avoid calling the function again until the credentials have expired (if it is None, then the credentials will be interpreted as to never expire).

This enables using custom credential provisioning logic with cloud I/O scans, which allows for handling credential expiry.

Example for AWS

def get_credentials():
    import boto3

    session = boto3.Session(profile_name="profile2")
    creds = session.get_credentials()

    return {
        "aws_access_key_id": creds.access_key,
        "aws_secret_access_key": creds.secret_key,
        "aws_session_token": creds.token,
    }, None


lf = pl.scan_parquet(
    "s3://...",
    credential_provider=get_credentials,
)

Example for GCP

def get_credentials():
    from pathlib import Path

    import google.auth
    import google.auth.transport.requests
    import zoneinfo

    creds, _ = google.auth.load_credentials_from_file(
        Path.home() / ".config/gcloud/application_default_credentials.json"
    )

    auth_req = google.auth.transport.requests.Request()
    creds.refresh(auth_req)

    return {"bearer_token": creds.token}, int(
        creds.expiry.replace(
            # Google auth does not set this properly
            tzinfo=zoneinfo.ZoneInfo("UTC")
        ).timestamp()
    )


lf = pl.scan_parquet(
    "gs://...",
    credential_provider=get_credentials,
)

Azure

I don't have an Azure environment to test on, but the returned dictionary should contain { 'bearer_token': '...' }

Todos for follow-up PRs

• Add the option to other cloud-enabled scan_/read_ functions
• Add and link to a documentation page on the function return formats for different cloud types
• Use an automatic default function for when the parameter is not specified

[nameexhaustion]

I have moved PythonFunction from polars-plan/../python_udf.rs to polars-utils for re-use

[codecov]

Codecov Report

Attention: Patch coverage is 25.15337% with 366 lines in your changes missing coverage. Please review.

Project coverage is 80.00%. Comparing base (7472a76) to head (0cecce0).
Report is 31 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-io/src/cloud/credential_provider.rs	10.07%	339 Missing ⚠️
crates/polars-io/src/cloud/options.rs	51.85%	13 Missing ⚠️
crates/polars-utils/src/python_function.rs	84.61%	10 Missing ⚠️
py-polars/polars/io/parquet/functions.py	0.00%	2 Missing and 1 partial ⚠️
crates/polars-plan/src/dsl/expr_dyn_fn.rs	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #19271      +/-   ##
==========================================
- Coverage   80.11%   80.00%   -0.11%     
==========================================
  Files        1526     1528       +2     
  Lines      209338   209740     +402     
  Branches     2418     2419       +1     
==========================================
+ Hits       167707   167799      +92     
- Misses      41081    41390     +309     
- Partials      550      551       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.