GH-44308: [C++][FS][Azure] Implement SAS token authentication #45021

Tom-Newton · 2024-12-13T17:01:50Z

Rationale for this change

SAS token auth is sometimes useful and it the last one we haven't implemented.

What changes are included in this PR?

Implement ConfigureSasCredential
Update AzureOptions::FromUri so that simply appending a SAS token to a blob storage URI works. e.g. AzureOptions::FromUri("abfs://[email protected]/?se=2024-12-12T18:57:47Z&sig=pAs7qEBdI6sjUhqX1nrhNAKsTY%2B1SqLxPK%2BbAxLiopw%3D&sp=racwdxylti&spr=https,http&sr=c&sv=2024-08-04")
- SAS tokens are made up of a bunch of URI query parameters that I'm not sure we can exhaustively list.
- Therefore we now assume that any unrecognised URI query parameters are assumed to be part of a SAS token, instead of returning an error status.
Update CopyFile to use StartCopyFromUri instead of CopyFromUri
- This avoids the need to generate SAS tokens.
- Supports blobs bigger than 256MiB
- This makes [C++][FS][Azure] Test CopyFile() with non account key credential #41315 redundant

Are these changes tested?

Yes

Added new tests for authenticating with SAS and doing some operations including CopyFile
Added new tests for AzureOptions::FromUri with a SAS token.

I also made sure to run the tests which connect to real blob storage.

Are there any user-facing changes?

SAS token in now supported
Unrecognised URI query parameters are ignored by AzureOptions::FromUri instead of failing fast. IMO this is a regression but still the best option to support SAS token.

GitHub Issue: [C++][FS][Azure] SAS token authentication #44308

…n. This avoids cheating by using the account key again to generate SAS tokens in tests

…one on the fly.

kou · 2024-12-14T02:10:38Z

cpp/src/arrow/filesystem/azurefs.cc

+      // Assume these are part of a SAS token. Its not ideal to make such an assumption
+      // but given that a SAS token is a complex set of URI parameters, that could be
+      // tricky to exhaustively list I think its the best option.
+      credential_kind = CredentialKind::kSasToken;


We don't have the SAS token specification that includes parameter names used by a SAS token, right?

Yeah, I had a quick search and couldn't find what we need. If you think it's important I can try a bit harder. The closest I found seemed to be unabbreviated versions of what actually appears in the sas token.

I found it: https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#construct-a-user-delegation-sas

There are many parameters but can we check them...?

I think that doc is only for user delegated SAS tokens so I unioned it with the parameters for account and service SAS tokens and hopefully the spec is slowly changing.

I wasn't really confident on the best way to define a constant set of strings to do contains checks against in C++. Since there are only 27 values, I ended up with a constexpr array and std::find but please let me know if this is not a good option.

cpp/src/arrow/filesystem/azurefs.h

cpp/src/arrow/filesystem/azurefs.cc

Tom-Newton · 2024-12-14T12:56:10Z

Wait... I might have just accidentally worked out how to avoid any of the special authentication stuff for copying...

cpp/src/arrow/filesystem/azurefs.cc

cpp/src/arrow/filesystem/azurefs_test.cc

cpp/src/arrow/filesystem/azurefs.cc

kou

+1

kou · 2024-12-17T07:32:10Z

cpp/src/arrow/filesystem/azurefs.cc

@@ -147,6 +159,10 @@ Status AzureOptions::ExtractFromUriQuery(const Uri& uri) {
    } else if (kv.first == "background_writes") {
      ARROW_ASSIGN_OR_RAISE(background_writes,
                            ::arrow::internal::ParseBoolean(kv.second));
+    } else if (std::find(sas_token_query_parameters.begin(),


Can we use std::binary_search() with sorted sas_token_query_parameters?

https://en.cppreference.com/w/cpp/algorithm/binary_search

Could do but I would be a bit concerned about keeping sas_token_query_parameters sorted. It looks like C++20 allows using std::sort with constexpr but I believe arrow currently uses C++17. If you are concerned about the complexity of the lookup I think my preference would be to use a std::set and forget about trying to make it a constexpr.

std::set is OK.

Tom-Newton added 10 commits December 11, 2024 18:28

Working with ConfigureSasCredential

1abba8e

Real access test working against azurite

ea091b4

Fix creating getting Azure env in test

6df79f7

Use credential_kind_ to detect credential type inside GenerateSASToke…

567488b

…n. This avoids cheating by using the account key again to generate SAS tokens in tests

Use original sas token inside CopyFile instead of generating a new …

6869067

…one on the fly.

Fix appending SAS token twice - all tests pass

0ffd832

Support SAS token in AzureOptions::FromUri

a2b826c

Update SasCredential test to cover extra query parmaters

9c6ec0e

Adjust comment and increase SAS expiry time

f4177be

Update comments

d19e87f

github-actions bot added Component: C++ awaiting review Awaiting review labels Dec 13, 2024

Autoformat

15fe2c6

Tom-Newton marked this pull request as ready for review December 13, 2024 17:07

kou reviewed Dec 14, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Dec 14, 2024

PR comments

b57ebea

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 14, 2024

Tom-Newton marked this pull request as draft December 14, 2024 12:56

Tom-Newton added 5 commits December 14, 2024 13:39

Avoid generating SAS tokens on the fly.

e200b8e

Add an extra test copying to a different container

b0fd1d3

Update comment

744e111

Remove AzureOptions::GenerateSASToken

c8fd94d

Remove debug change

83a0b40

Tom-Newton marked this pull request as ready for review December 14, 2024 14:22

kou reviewed Dec 15, 2024

View reviewed changes

cpp/src/arrow/filesystem/azurefs.cc Outdated Show resolved Hide resolved

cpp/src/arrow/filesystem/azurefs_test.cc Outdated Show resolved Hide resolved

cpp/src/arrow/filesystem/azurefs.cc Show resolved Hide resolved

cpp/src/arrow/filesystem/azurefs.cc Show resolved Hide resolved

github-actions bot removed the awaiting change review Awaiting change review label Dec 15, 2024

github-actions bot added the awaiting changes Awaiting changes label Dec 15, 2024

Tom-Newton added 3 commits December 15, 2024 09:42

Rename Sas -> SAS

d60742b

Add a comment about the polling interval

63f9773

Only allow query parameters we know can be part of a sas token

d55510a

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 15, 2024

More renaming Sas -> SAS

7576c4c

Tom-Newton force-pushed the tomnewton/implement_sas_auth_on_azure/GH-44308 branch from d30e6ce to 7576c4c Compare December 15, 2024 11:15

kou approved these changes Dec 17, 2024

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Dec 17, 2024

Give up on contexpr. Make it a static const std::set<std::string>

eece05e

kou merged commit ba2b9e5 into apache:main Dec 17, 2024
37 of 38 checks passed

kou removed the awaiting merge Awaiting merge label Dec 17, 2024

kou mentioned this pull request Dec 17, 2024

[C++][FS][Azure] SAS token authentication #44308

Closed

github-actions bot added the awaiting changes Awaiting changes label Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-44308: [C++][FS][Azure] Implement SAS token authentication #45021

GH-44308: [C++][FS][Azure] Implement SAS token authentication #45021

Tom-Newton commented Dec 13, 2024 •

edited

Loading

kou Dec 14, 2024

Tom-Newton Dec 14, 2024

kou Dec 15, 2024

Tom-Newton Dec 15, 2024

Tom-Newton commented Dec 14, 2024

kou left a comment

kou Dec 17, 2024

Tom-Newton Dec 17, 2024 •

edited

Loading

kou Dec 17, 2024

GH-44308: [C++][FS][Azure] Implement SAS token authentication #45021

GH-44308: [C++][FS][Azure] Implement SAS token authentication #45021

Conversation

Tom-Newton commented Dec 13, 2024 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

kou Dec 14, 2024

Choose a reason for hiding this comment

Tom-Newton Dec 14, 2024

Choose a reason for hiding this comment

kou Dec 15, 2024

Choose a reason for hiding this comment

Tom-Newton Dec 15, 2024

Choose a reason for hiding this comment

Tom-Newton commented Dec 14, 2024

kou left a comment

Choose a reason for hiding this comment

kou Dec 17, 2024

Choose a reason for hiding this comment

Tom-Newton Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

kou Dec 17, 2024

Choose a reason for hiding this comment

Tom-Newton commented Dec 13, 2024 •

edited

Loading

Tom-Newton Dec 17, 2024 •

edited

Loading