Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

config: add iceberg_rest_catalog_authentication_mode #24986

Merged
merged 6 commits into from
Feb 22, 2025
Merged

config: add iceberg_rest_catalog_authentication_mode #24986

merged 6 commits into from
Feb 22, 2025

Conversation

WillemKauf
Copy link
Contributor

@WillemKauf WillemKauf commented Jan 30, 2025 •
edited by dotnwat
Loading

Add iceberg_rest_catalog_authentication_mode and iceberg_rest_catalog_oauth2_server_uri configuration options.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@WillemKauf WillemKauf requested a review from a team as a code owner January 30, 2025 22:17
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Jan 31, 2025 •
edited
Loading

Retry command for Build#61413

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":false}
tests/rptest/tests/polaris_catalog_smoke_test.py::PolarisCatalogSmokeTest.test_connecting_to_catalog@{"cloud_storage_type":1,"with_tls":true}

@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch 2 times, most recently from f1ecafb to 819b3f9 Compare January 31, 2025 02:24
@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from 819b3f9 to 2b38389 Compare January 31, 2025 02:56
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Jan 31, 2025 •
edited
Loading

CI test results

test results on build#61423
test_id test_kind job_url test_status passed
gtest_raft_rpunit.gtest_raft_rpunit unit https://buildkite.com/redpanda/redpanda/builds/61423#0194ba4c-385f-48a1-bd80-40a25c387208 FLAKY 1/2
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery ducktape https://buildkite.com/redpanda/redpanda/builds/61423#0194baa8-73b0-4829-b1fe-6bd344b0b466 FLAKY 1/2
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery ducktape https://buildkite.com/redpanda/redpanda/builds/61423#0194baac-12d6-4de7-be47-f79878594dbc FLAKY 1/3
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/61423#0194baac-12d7-4f9b-b2b5-f68739b0ee24 FLAKY 1/2
rptest.tests.partition_force_reconfiguration_test.NodeWiseRecoveryTest.test_node_wise_recovery.dead_node_count=1 ducktape https://buildkite.com/redpanda/redpanda/builds/61423#0194baa8-73b0-4829-b1fe-6bd344b0b466 FLAKY 1/2
test results on build#61438
test_id test_kind job_url test_status passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery ducktape https://buildkite.com/redpanda/redpanda/builds/61438#0194bce7-8113-4b07-bac1-0261019fff89 FLAKY 1/2
rptest.tests.consumer_group_balancing_test.ConsumerGroupBalancingTest.test_coordinator_nodes_balance ducktape https://buildkite.com/redpanda/redpanda/builds/61438#0194bce7-8115-4648-bc9a-a9632dc84135 FLAKY 1/2
rptest.tests.datalake.simple_connect_test.RedpandaConnectIcebergTest.test_translating_avro_serialized_records.cloud_storage_type=CloudStorageType.S3.scenario=simple ducktape https://buildkite.com/redpanda/redpanda/builds/61438#0194bce3-b648-4de6-81d6-0a94cf40d184 FLAKY 1/2
rptest.tests.internal_topic_protection_test.InternalTopicProtectionLargeClusterTest.test_consumer_offset_topic ducktape https://buildkite.com/redpanda/redpanda/builds/61438#0194bce7-8115-4648-bc9a-a9632dc84135 FLAKY 1/2
rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/61438#0194bce7-8115-4648-bc9a-a9632dc84135 FLAKY 1/2
test results on build#61474
test_id test_kind job_url test_status passed
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade ducktape https://buildkite.com/redpanda/redpanda/builds/61474#0194bf68-19f3-443a-aa71-3c6360d84182 FLAKY 1/2
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_topic_lifecycle.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=True ducktape https://buildkite.com/redpanda/redpanda/builds/61474#0194bf71-acf6-446d-aebf-c791682c9ad4 FLAKY 1/2
test results on build#61885
test_id test_kind job_url test_status passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery ducktape https://buildkite.com/redpanda/redpanda/builds/61885#01950775-b1d7-4ed7-9126-77a442974ed7 FLAKY 1/6
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery ducktape https://buildkite.com/redpanda/redpanda/builds/61885#01950789-96c8-4bcc-ad21-698235ef7662 FLAKY 1/4
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3 ducktape https://buildkite.com/redpanda/redpanda/builds/61885#01950775-b1d7-4ed7-9126-77a442974ed7 FLAKY 1/3
rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/61885#01950775-b1d7-4ed7-9126-77a442974ed7 FLAKY 1/2
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic ducktape https://buildkite.com/redpanda/redpanda/builds/61885#01950775-b1d7-4ed7-9126-77a442974ed7 FLAKY 1/2
test results on build#61888
test_id test_kind job_url test_status passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery ducktape https://buildkite.com/redpanda/redpanda/builds/61888#01950a31-e3bb-4de6-92a1-63ed11069502 FLAKY 1/2
rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True ducktape https://buildkite.com/redpanda/redpanda/builds/61888#01950a2e-ca93-4c66-b19f-8174742d626b FLAKY 1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=10 ducktape https://buildkite.com/redpanda/redpanda/builds/61888#01950a31-e3bb-4de6-92a1-63ed11069502 FLAKY 1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3 ducktape https://buildkite.com/redpanda/redpanda/builds/61888#01950a31-e3bb-4de6-92a1-63ed11069502 FLAKY 1/2
rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/61888#01950a31-e3bb-4de6-92a1-63ed11069502 FLAKY 1/3
test results on build#62090
test_id test_kind job_url test_status passed
rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete ducktape https://buildkite.com/redpanda/redpanda/builds/62090#019526f9-0882-4256-9aa4-e43b45a59c63 FLAKY 1/2

@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch 2 times, most recently from 1a892e9 to bb87125 Compare January 31, 2025 13:42
dotnwat
dotnwat reviewed Jan 31, 2025
View reviewed changes
@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from bb87125 to a5b738a Compare January 31, 2025 21:33
@WillemKauf
Copy link
Contributor Author

Force push to:

  • Move commit history around for config properties
  • Squash commit for rest_catalog_test

@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from a5b738a to 84e8449 Compare February 1, 2025 00:32
@WillemKauf
Copy link
Contributor Author

Force push to:

  • change maybe_make_credentials_or_token() to make_credentials_and_token()

@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from 84e8449 to 493a2a2 Compare February 1, 2025 01:02
asimms41
asimms41 reviewed Feb 7, 2025
View reviewed changes
@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch 3 times, most recently from f0e5026 to 3dced9a Compare February 15, 2025 01:14
@WillemKauf
Copy link
Contributor Author

Force push to:

  • Rebase to upstream/dev to fix merge conflicts

@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from 3dced9a to d4456d3 Compare February 15, 2025 13:58
@WillemKauf
Copy link
Contributor Author

Force push to:

  • Make changes to config/configuration.cc as requested by docs team.

@andrwng andrwng self-requested a review February 20, 2025 19:07
@dotnwat
Copy link
Member

dotnwat commented Feb 20, 2025

is this ready for review, i can't tell

@WillemKauf
Copy link
Contributor Author

is this ready for review, i can't tell

Yes

andrwng
andrwng reviewed Feb 21, 2025
View reviewed changes
Copy link
Contributor

@andrwng andrwng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

src/v/config/configuration.cc Outdated
Comment on lines 3835 to 3837
"from the URI specified by `iceberg_rest_catalog_oauth2_server_uri`. If "
"this server URI is left undefined, the deprecated Iceberg catalog "
"endpoint `/v1/oauth/tokens` is used instead.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this server URI is left undefined, the deprecated Iceberg catalog endpoint /v1/oauth/tokens is used instead.

nit: maybe remove this bit to avoid duplicating, given it's already covered in the description of that property

@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from d4456d3 to 48760be Compare February 21, 2025 04:11
@WillemKauf
Copy link
Contributor Author

Force push to:

  • Change std::optional<domain_error> to expected<std::monostate> in catalog_client::maybe_add_bearer_auth()
  • Add invalid test cases to cluster_config_test.py
  • Clean up commit history by squashing some commits together

@WillemKauf
Copy link
Contributor Author

I'm going to leave parameterized Nessie testing as work for a follow-up PR.

@andrwng andrwng self-requested a review February 21, 2025 21:45
@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from 48760be to 6da2082 Compare February 21, 2025 22:12
@WillemKauf
Copy link
Contributor Author

Force push to:

  • Remove oauth/tokens suffix appending to oauth2_server_uri. Users must specified the full token path to iceberg_rest_catalog_oauth2_server_uri.

@WillemKauf
config: add iceberg_rest_catalog_oauth2_server_uri

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
0e908b8 
For allowing the user to specify a uri to an OAuth2 server
for use in authenticating requests made by the `catalog_client`
to the catalog.
@WillemKauf
config: add iceberg_rest_catalog_authentication_mode

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
099f5c6 
For configuring the authentication method used for the `catalog_client`
requests made to the catalog.

There are three possible values for this `enum`:

  1. `::none`, where no authentication is attempted.
  2. `::bearer`, where the token provided via `iceberg_rest_catalog_token`
      is used for authentication.
  3. `::oauth2`, where the client credentials defined via
     `iceberg_rest_catalog_client_id/secret` are used to retrieve
      an access token for future use.
@WillemKauf
iceberg: add oauth2_server_uri to credentials

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
fc6c796
@WillemKauf
datalake: add make_credentials_or_token()

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
0113b50 
Depending on authentication settings, we may either construct
user credentials for use with the OAuth2 endpoint for retrieving a
token, construct the token directly, or do nothing at all.
@WillemKauf
iceberg: use oauth2_server_uri in catalog_client
8dcfaa2
@WillemKauf
iceberg: add maybe_add_bearer_auth() in catalog_client
c7aded0
@WillemKauf WillemKauf force-pushed the iceberg_authentication_properties branch from 6da2082 to c7aded0 Compare February 22, 2025 00:20
@WillemKauf
Copy link
Contributor Author

Force push to:

  • Update configuration descriptions

@WillemKauf WillemKauf enabled auto-merge February 22, 2025 02:00
@WillemKauf WillemKauf merged commit c93a0bf into redpanda-data:dev Feb 22, 2025
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dotnwat dotnwat

@asimms41 asimms41

@andrwng andrwng

Assignees
No one assigned
Labels
area/build area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@WillemKauf @vbotbuildovich @dotnwat @andrwng @asimms41