Auth Manager API part 1: HTTPRequest, HTTPHeader #11769

adutra · 2024-12-12T18:19:42Z

As requested, I'm splitting #10753 in many PRs. This one is the first one. It introduces HTTPRequest which is a prerequisite for the AuthManager API.

adutra · 2024-12-12T18:23:05Z

core/src/main/java/org/apache/iceberg/rest/HTTPRequest.java

+
+  /** Returns the headers of this request. */
+  @Value.Default
+  default HTTPHeaders headers() {


@danielcweeks @nastra FYI, I wasn't satisfied with returning Map<String, List<String>> from here, especially because of corner cases around case-sensitivity. I was planning to tackle that as a follow-up task, but since we are starting from square zero here, what you see here is the "enhanced" version.

adutra · 2024-12-12T18:25:08Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+   * values. Header names are case-insensitive.
+   */
+  @Value.Lazy
+  default Map<String, List<String>> asMap() {


All these methods will be useful later.

adutra · 2024-12-12T18:25:43Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+@Value.Style(depluralize = true)
+@Value.Immutable
+@SuppressWarnings({"ImmutablesStyle", "SafeLoggingPropagation"})
+public interface HTTPHeaders {


This class is loosely inspired by org.apache.hc.core5.http.message.HeaderGroup.

nastra · 2024-12-13T08:32:29Z

core/src/main/java/org/apache/iceberg/rest/HTTPRequest.java

+   */
+  @Value.Lazy
+  default URI requestUri() {
+    return RESTUtil.buildRequestUri(this);


to me it seems like we should rather embed the functionality of building an URI from a HTTPRequest here (and also do the same for encoding the body). Additionally, building an URI from the path does some validation to make sure it doesn't start with a /. I'd say this is something that we can move into a check method annotated with @Value.Check. That way we don't need the util methods in RESTUtil and can move testing into TestHTTPRequest. I've added an example diff below:

--- a/core/src/main/java/org/apache/iceberg/rest/HTTPRequest.java +++ b/core/src/main/java/org/apache/iceberg/rest/HTTPRequest.java @@ -18,10 +18,14 @@ */ package org.apache.iceberg.rest; +import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; import java.net.URI; +import java.net.URISyntaxException; import java.util.Map; import javax.annotation.Nullable; +import org.apache.hc.core5.net.URIBuilder; +import org.apache.iceberg.exceptions.RESTException; import org.immutables.value.Value; /** Represents an HTTP request. */ @@ -49,7 +53,19 @@ public interface HTTPRequest { */ @Value.Lazy default URI requestUri() { - return RESTUtil.buildRequestUri(this); + // if full path is provided, use the input path as path + String fullPath = + (path().startsWith("https://") || path().startsWith("http://")) + ? path() + : String.format("%s/%s", baseUri(), path()); + try { + URIBuilder builder = new URIBuilder(RESTUtil.stripTrailingSlash(fullPath)); + queryParameters().forEach(builder::addParameter); + return builder.build(); + } catch (URISyntaxException e) { + throw new RESTException( + "Failed to create request URI from base %s, params %s", fullPath, queryParameters()); + } } /** Returns the HTTP method of this request. */ @@ -77,7 +93,17 @@ public interface HTTPRequest { @Nullable @Value.Redacted default String encodedBody() { - return RESTUtil.encodeRequestBody(this); + Object body = body(); + if (body instanceof Map) { + return RESTUtil.encodeFormData((Map<?, ?>) body); + } else if (body != null) { + try { + return mapper().writeValueAsString(body); + } catch (JsonProcessingException e) { + throw new RESTException(e, "Failed to encode request body: %s", body); + } + } + return null; } /** @@ -88,4 +114,13 @@ public interface HTTPRequest { default ObjectMapper mapper() { return RESTObjectMapper.mapper(); } + + @Value.Check + default void check() { + if (path().startsWith("/")) { + throw new RESTException( + "Received a malformed path for a REST request: %s. Paths should not start with /", + path()); + } + } }

Yes that's a good idea.

nastra · 2024-12-13T08:39:19Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+   * case-insensitive.
+   */
+  @Value.Lazy
+  default ListMultimap<String, String> asMultiMap() {


do we actually have a use case for this? I see the value in having asMap() / asSimpleMap() but I feel like we don't need this method atm?

This one will be very useful later to do things like this in HTTPClient:

HTTPRequest req = ... HttpUriRequestBase request = new HttpUriRequestBase(req.method().name(), req.requestUri()); req.headers().asMultiMap().forEach(request::addHeader);

nastra · 2024-12-13T08:41:46Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+    return builder.build();
+  }
+
+  static HTTPHeaders fromMultiMap(Multimap<String, String> headers) {


same as I mentioned above. Do we actually have a use case for a bunch of these static methods? I would try to keep those to the amount we actually need

If the methods are tested, is that a big deal? This one indeed has no usage today, but more auth managers in the future could use it.

The Iceberg project typically tries to only add things to APIs that are needed for current use cases instead of planning for potential use cases that may or may not actually exist/happen (independent of whether there are tests for unused methods or not).
You also have to keep in mind that everything that's being added to iceberg-core can't be easily changed without going through an API deprecation cycle, so we're trying to be mindful about what's really required in terms of APIs

Fair enough, let's remove all the static methods for now.

nastra · 2024-12-13T08:44:31Z

core/src/test/java/org/apache/iceberg/rest/TestHTTPHeaders.java

+
+  final HTTPHeaders headers =
+      HTTPHeaders.of(
+          HTTPHeader.of("header1", "value1a"),


please also add a test where the key/value is null

nastra · 2024-12-13T08:44:38Z

core/src/test/java/org/apache/iceberg/rest/TestHTTPHeaders.java

+
+class TestHTTPHeaders {
+
+  final HTTPHeaders headers =


can be private

nastra · 2024-12-13T08:46:48Z

core/src/test/java/org/apache/iceberg/rest/TestHTTPHeaders.java

+
+  @Test
+  void entries() {
+    assertThat(headers.entries("header1"))


please also add tests where null is passed to all the methods that take nullable parameters, such as entries()/contains()/addIfAbsent()

nastra · 2024-12-13T11:41:41Z

core/src/test/java/org/apache/iceberg/rest/TestHTTPHeaders.java

@@ -95,6 +100,9 @@ void addIfAbsentHTTPHeader() {
                "header1", List.of("value1a", "value1b"),
                "header2", List.of("value2"),
                "header3", List.of("value3")));
+
+    assertThatThrownBy(() -> headers.addIfAbsent((HTTPHeaders) null))
+        .isInstanceOf(NullPointerException.class);


checks like this should always have a .hasMessage check to make sure we get the right error msg back (unfortunately we can't easily enforce such a rule via checkstyle)

The message is generated by Immutables and will be rather cryptic. But OK.

nastra · 2024-12-13T12:14:32Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+   * Returns a new instance with the added header, or the current instance if the header is already
+   * present.
+   */
+  default HTTPHeaders addIfAbsent(HTTPHeader header) {


we might want to consider naming this (and the other method) withHeaderIfAbsent() to clearly indicate that (potentially) a new instance is returned because add in the name suggests that the existing object is modified

nastra · 2024-12-13T12:22:00Z

core/src/test/java/org/apache/iceberg/rest/TestHTTPRequest.java

+  @ParameterizedTest
+  @MethodSource("validRequestBodies")
+  public void encodedBody(HTTPRequest request, String expected) {
+    assertThat(request.encodedBody()).isEqualTo(expected);


personally I think the test code would be much more readable without having a parameterized test and rather have just two separate test methods that verify the body is properly encoded as form data/JSON, but I'm ok with what's currently here and I'll leave that up to you.

Also we probably might want to have some tests where the body can't be encoded due to:

body is null

body is a Map with null values

body is an invalid JSON

nastra

LGTM, thanks @adutra for all your work on this.

danielcweeks · 2024-12-16T17:59:25Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+  HTTPHeaders EMPTY = of();
+
+  /** Returns all the header entries in this group. */
+  List<HTTPHeader> entries();


Shouldn't this be Set<HTTPHeader>? This would also take care of cases where there are actualy k/v duplicates (which we don't appear to handle)

I feel like we should also annotate this as @NotNull

I don't see a @NotNull annotation available in iceberg-core? Besides, the immutable generated class is annotated with @AllParametersAreNonNullByDefault.

@danielcweeks @NotNull shouldn't be needed here, because everything not marked as @Nullable or Optional is implicitly not null with Immutables

danielcweeks · 2024-12-16T17:59:37Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+  List<HTTPHeader> entries();
+
+  /** Returns all the entries in this group for the given name (case-insensitive). */
+  default List<HTTPHeader> entries(String name) {


Same here, set

danielcweeks · 2024-12-16T18:02:27Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+   * Returns a new instance with the added headers, or the current instance if all headers are
+   * already present.
+   */
+  default HTTPHeaders withHeaderIfAbsent(HTTPHeaders headers) {


I feel like putIfAbsent() is more idiomatic here. Otherwise it just code would generally be redundant:

headers.putHeaderIfAbsent(header) vs. headers.putIfAbsent(header)

danielcweeks · 2024-12-16T18:03:50Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+   * Returns a new instance with the added header, or the current instance if the header is already
+   * present.
+   */
+  default HTTPHeaders withHeaderIfAbsent(HTTPHeader header) {


Same idiomatic comment here. Generally we use with prefix for builders, not mutators.

danielcweeks · 2024-12-16T18:08:14Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+  }
+
+  /** Returns whether this group contains an entry with the given name (case-insensitive). */
+  default boolean contains(String name) {


Is there are reason we're making this overridable? It seems like we should insulate the case behavior by making this a private static method.

danielcweeks · 2024-12-16T18:11:30Z

core/src/main/java/org/apache/iceberg/rest/HTTPHeaders.java

+
+    String name();
+
+    @Value.Redacted


Does this mean that we redact all header values? Maybe I don't fully understand the Immutable behavior here, but that's a bit problematic because we probably only want to redact sensitive headers (like AUTHORIZATION).

danielcweeks · 2024-12-16T18:14:51Z

core/src/test/java/org/apache/iceberg/rest/TestHTTPHeaders.java

+import org.apache.iceberg.rest.HTTPHeaders.HTTPHeader;
+import org.junit.jupiter.api.Test;
+
+class TestHTTPHeaders {


It would be good to add a test for duplicate behavior as well.

Auth Manager API part 1: HTTPRequest, HTTPHeader

7db53d8

github-actions bot added the core label Dec 12, 2024

adutra commented Dec 12, 2024

View reviewed changes

adutra mentioned this pull request Dec 12, 2024

REST: AuthManager API #10753

Closed

adutra commented Dec 12, 2024

View reviewed changes

nastra reviewed Dec 13, 2024

View reviewed changes

nastra requested a review from danielcweeks December 13, 2024 09:01

review

e486e0d

nastra reviewed Dec 13, 2024

View reviewed changes

adutra added 3 commits December 13, 2024 12:44

remove static methods

0449744

verify error messages

d9a05fb

checkstyle

4c0e650

nastra reviewed Dec 13, 2024

View reviewed changes

review

56ff7b2

nastra approved these changes Dec 13, 2024

View reviewed changes

danielcweeks reviewed Dec 16, 2024

View reviewed changes

review

ca6eb1e

danielcweeks reviewed Dec 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auth Manager API part 1: HTTPRequest, HTTPHeader #11769

Auth Manager API part 1: HTTPRequest, HTTPHeader #11769

adutra commented Dec 12, 2024

adutra Dec 12, 2024

adutra Dec 12, 2024

adutra Dec 12, 2024

nastra Dec 13, 2024

adutra Dec 13, 2024

nastra Dec 13, 2024

adutra Dec 13, 2024

nastra Dec 13, 2024

adutra Dec 13, 2024

nastra Dec 13, 2024

adutra Dec 13, 2024

nastra Dec 13, 2024

nastra Dec 13, 2024

nastra Dec 13, 2024

nastra Dec 13, 2024

adutra Dec 13, 2024

nastra Dec 13, 2024

nastra Dec 13, 2024

nastra left a comment

danielcweeks Dec 16, 2024 •

edited

Loading

danielcweeks Dec 16, 2024

adutra Dec 16, 2024

nastra Dec 16, 2024

danielcweeks Dec 16, 2024

danielcweeks Dec 16, 2024

danielcweeks Dec 16, 2024

danielcweeks Dec 16, 2024

danielcweeks Dec 16, 2024

danielcweeks Dec 16, 2024

Auth Manager API part 1: HTTPRequest, HTTPHeader #11769

Are you sure you want to change the base?

Auth Manager API part 1: HTTPRequest, HTTPHeader #11769

Conversation

adutra commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nastra left a comment

Choose a reason for hiding this comment

danielcweeks Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielcweeks Dec 16, 2024 •

edited

Loading