Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile #7329

Open
wants to merge 19 commits into
base: HADOOP-19256-s3-conditional-writes
Choose a base branch
from
Open

HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile #7329

wants to merge 19 commits into from

Conversation

saikatroy038
Copy link
Contributor

@saikatroy038 saikatroy038 commented Jan 27, 2025 •
edited
Loading

Description of PR

Cherry-pick of #7011, and additional changes on top of it

How was this patch tested?

integration tests ran on ap-south-1
Failing tests:
ITestAwsSdkWorkarounds.testNoisyLogging
ITestConnectionTimeouts.testObjectUploadTimeouts

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+1 💚 mvninstall 39m 2s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 0m 42s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 36s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 33s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 0m 39s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 0m 41s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 33s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 9s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 37m 7s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 30s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 27s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 27s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 22s /results-checkstyle-hadoop-tools_hadoop-aws.txt hadoop-tools/hadoop-aws: The patch generated 71 new + 5 unchanged - 0 fixed = 76 total (was 5)
+1 💚 mvnsite 0m 31s the patch passed
-1 ❌ javadoc 0m 28s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ javadoc 0m 25s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 spotbugs 1m 6s the patch passed
+1 💚 shadedclient 38m 0s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 47s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
129m 9s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 506c09f684c8 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 29ff04a
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/testReport/
Max. process+thread count 529 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 50s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+1 💚 mvninstall 39m 52s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 0m 44s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 35s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 32s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 0m 41s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 0m 41s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 33s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 8s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 52s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 29s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 21s /results-checkstyle-hadoop-tools_hadoop-aws.txt hadoop-tools/hadoop-aws: The patch generated 70 new + 5 unchanged - 0 fixed = 75 total (was 5)
+1 💚 mvnsite 0m 34s the patch passed
-1 ❌ javadoc 0m 29s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ javadoc 0m 25s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 spotbugs 1m 6s the patch passed
+1 💚 shadedclient 37m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 44s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
131m 25s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 88b6aaed7d3f 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / aa1defc
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/testReport/
Max. process+thread count 641 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran requested changes Feb 5, 2025
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catching up with this.

I can see it is still part way through doing those first changes I'd suggested. I don't think we need that map of options, I'd rather they get extracted.

we have two create options then

  • reject overwrite
  • later: require match + etag

the code is very, very similar, all that is different is that the second one needs an etag

we need to think of the best way to get both of these down.

For testing large files, I also want to add an option for "always do a multipart write" -but that can go in as a header with a new option defined.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java
.multipartUpload(CompletedMultipartUpload.builder().parts(partETags).build());
if (putOptions.isconditionalPutEnabled()) {
requestBuilder.overrideConfiguration(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is from the first patch, isn't it? I'll have to look at my comments there

@steveloughran steveloughran self-assigned this Feb 6, 2025
@steveloughran steveloughran mentioned this pull request Feb 6, 2025
Draft
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+1 💚 mvninstall 40m 46s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 0m 45s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 34s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 32s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 0m 41s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 0m 40s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 32s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 9s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 29s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 21s /results-checkstyle-hadoop-tools_hadoop-aws.txt hadoop-tools/hadoop-aws: The patch generated 65 new + 5 unchanged - 0 fixed = 70 total (was 5)
+1 💚 mvnsite 0m 31s the patch passed
-1 ❌ javadoc 0m 29s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ javadoc 0m 26s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 spotbugs 1m 6s the patch passed
+1 💚 shadedclient 37m 54s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 45s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
132m 1s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 3212193e2d22 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / b19da55
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/testReport/
Max. process+thread count 528 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran and others added 2 commits February 7, 2025 10:00
@steveloughran
HADOOP-19256. review of conditional write
75544d0 
+ hadoop common and s3a create file options

wired up to s3a create file builder, and nowhere else.

Change-Id: Ieaafa9ecdfd68306deb41ecf40c416374a310859
HADOOP-19256: add tests
26811e6
@github-actions github-actions bot added the Common label Feb 7, 2025
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 55s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 20s Maven dependency ordering for branch
+1 💚 mvninstall 35m 11s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 18m 44s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 17m 23s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 4m 41s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 34s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 5s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 38s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 3m 51s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 39m 30s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 1m 27s the patch passed
+1 💚 compile 18m 15s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 18m 15s the patch passed
+1 💚 compile 17m 10s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 17m 10s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 26s /results-checkstyle-root.txt root: The patch generated 133 new + 23 unchanged - 0 fixed = 156 total (was 23)
+1 💚 mvnsite 2m 30s the patch passed
-1 ❌ javadoc 0m 50s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
-1 ❌ javadoc 0m 46s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
+1 💚 spotbugs 4m 10s the patch passed
+1 💚 shadedclient 38m 42s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 38s hadoop-common in the patch passed.
-1 ❌ unit 2m 55s /patch-unit-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
252m 37s
Reason Tests
Failed junit tests hadoop.fs.s3a.impl.TestRequestFactory
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 23f3f779377a 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 26811e6
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/testReport/
Max. process+thread count 2734 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran and others added 2 commits February 13, 2025 12:57
@steveloughran
HADOOP-19256. filesystem options for overwrite, etag and small multip…
5bfbeec 
…art files

createFile() options are passed down all the way to object creation
* fs.option.create.conditional.overwrite: sets the header.
  MUST be in close(). After all, create(overwrite=false) is eager.
* fs.option.create.conditional.overwrite.etag requests etag writes.
  MAY be in create(); may be in close().
* fs.option.create.content.type for mime type.
* fs.option.create.in.close to ask for create in close, as fs capability.
  (maybe make this an fs path capability only?)
* fs.s3a.create.multipart : allows tests to create small multipart files

Javadocs of the fs.option.create try to define semantics; will need
strict fs specification soon.

Bool params come down in an enumset of flags; I'm going to do that in
more code as it is more flexible over time than many booleans.

- etag values are passed down but not wired up/tested
- content type flag exists but is ignored

New WriteObjectFlags enum is in new package o.a.h.fs.s3a.write;
to match the streams package in another ongoing PR.
Goal, as we maintain things, all code related to writing can
go in here.

also: options set by WriteObjectFlags return true in hasCapability()
of the stream

Change-Id: I301abd7397accbd278d05f42f858223ba1349fc8
HADOOP-19256: Use forced multipart in tests, add If-Match header in r…
3e59b74 
…equest.
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 28s Maven dependency ordering for branch
+1 💚 mvninstall 34m 19s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 19m 15s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 18m 2s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 26s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 27s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 1m 58s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 30s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 49s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 37m 58s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 1m 25s the patch passed
+1 💚 compile 17m 55s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 17m 55s the patch passed
+1 💚 compile 17m 6s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 17m 6s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 31s /results-checkstyle-root.txt root: The patch generated 142 new + 23 unchanged - 0 fixed = 165 total (was 23)
+1 💚 mvnsite 2m 26s the patch passed
-1 ❌ javadoc 1m 10s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 50s /patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 46s /patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
+1 💚 spotbugs 4m 9s the patch passed
+1 💚 shadedclient 39m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 35s hadoop-common in the patch passed.
+1 💚 unit 2m 55s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
250m 42s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 9d450b01ee8e 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 3e59b74
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/testReport/
Max. process+thread count 1255 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 5m 57s Maven dependency ordering for branch
+1 💚 mvninstall 34m 52s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 18m 43s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 17m 12s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 33s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 30s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 6s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 38s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 50s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 1m 23s the patch passed
+1 💚 compile 17m 42s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 17m 42s the patch passed
+1 💚 compile 17m 14s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 17m 14s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 25s /results-checkstyle-root.txt root: The patch generated 244 new + 23 unchanged - 0 fixed = 267 total (was 23)
+1 💚 mvnsite 2m 29s the patch passed
-1 ❌ javadoc 1m 10s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 50s /patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 45s /patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
-1 ❌ spotbugs 1m 30s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 38m 53s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 40s hadoop-common in the patch passed.
+1 💚 unit 2m 59s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
250m 6s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1598]
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux aa90e18fae55 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 294b14d
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/testReport/
Max. process+thread count 1255 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 34s Maven dependency ordering for branch
-1 ❌ mvninstall 47m 43s /branch-mvninstall-root.txt root in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ compile 0m 31s /branch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt root in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ compile 0m 31s /branch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt root in HADOOP-19256-s3-conditional-writes failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
+1 💚 checkstyle 5m 47s HADOOP-19256-s3-conditional-writes passed
-1 ❌ mvnsite 0m 33s /branch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ mvnsite 0m 42s /branch-mvnsite-hadoop-tools_hadoop-aws.txt hadoop-aws in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ javadoc 0m 31s /branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 31s /branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 31s /branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
-1 ❌ spotbugs 1m 39s /branch-spotbugs-hadoop-common-project_hadoop-common.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ spotbugs 0m 31s /branch-spotbugs-hadoop-tools_hadoop-aws.txt hadoop-aws in HADOOP-19256-s3-conditional-writes failed.
+1 💚 shadedclient 13m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 44s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 49s /patch-mvninstall-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ mvninstall 0m 30s /patch-mvninstall-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ compile 1m 20s /patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javac 1m 20s /patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ compile 0m 13s /patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
-1 ❌ javac 0m 13s /patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 14s /buildtool-patch-checkstyle-root.txt The patch fails to run checkstyle in root
-1 ❌ mvnsite 0m 55s /patch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ mvnsite 0m 24s /patch-mvnsite-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ javadoc 1m 10s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 26s /patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 35s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06 with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06 generated 24 new + 0 unchanged - 0 fixed = 24 total (was 0)
-1 ❌ spotbugs 0m 24s /patch-spotbugs-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ spotbugs 0m 24s /patch-spotbugs-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ shadedclient 16m 45s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 27s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ unit 0m 32s /patch-unit-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ asflicense 0m 44s /results-asflicense.txt The patch generated 1 ASF License warnings.
88m 22s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux f7f7dd227502 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / b007664
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/testReport/
Max. process+thread count 84 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 18m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 13s Maven dependency ordering for branch
+1 💚 mvninstall 35m 10s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 18m 50s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 17m 17s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 39s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 32s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 4s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 36s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 50s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 27s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 1m 23s the patch passed
+1 💚 compile 18m 34s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 18m 34s the patch passed
+1 💚 compile 17m 16s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 17m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 31s /results-checkstyle-root.txt root: The patch generated 244 new + 23 unchanged - 0 fixed = 267 total (was 23)
+1 💚 mvnsite 2m 28s the patch passed
-1 ❌ javadoc 1m 11s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 1m 37s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
-1 ❌ spotbugs 1m 31s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 38m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 35s hadoop-common in the patch passed.
+1 💚 unit 2m 55s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 1s The patch does not generate ASF License warnings.
269m 8s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1598]
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 23195b073f37 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / b007664
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/testReport/
Max. process+thread count 2927 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 6s Maven dependency ordering for branch
+1 💚 mvninstall 34m 16s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 18m 52s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 17m 14s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 30s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 31s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 4s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 37s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 51s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 42s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 1m 33s the patch passed
+1 💚 compile 19m 22s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 19m 22s the patch passed
+1 💚 compile 19m 40s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 19m 40s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 32s /results-checkstyle-root.txt root: The patch generated 266 new + 23 unchanged - 0 fixed = 289 total (was 23)
+1 💚 mvnsite 5m 15s the patch passed
-1 ❌ javadoc 1m 9s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 1m 49s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
-1 ❌ spotbugs 1m 33s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 40m 44s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 24s hadoop-common in the patch passed.
+1 💚 unit 2m 58s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 0s The patch does not generate ASF License warnings.
260m 7s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1598]
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 5f43868e11e3 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 6a8e199
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/testReport/
Max. process+thread count 2140 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 50s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 5m 42s Maven dependency ordering for branch
+1 💚 mvninstall 36m 14s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 22m 50s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 21m 49s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 40s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 25s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 1m 52s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 28s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 50s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 40m 40s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 1m 26s the patch passed
+1 💚 compile 18m 16s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 18m 16s the patch passed
+1 💚 compile 17m 19s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 17m 19s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 32s /results-checkstyle-root.txt root: The patch generated 274 new + 23 unchanged - 0 fixed = 297 total (was 23)
+1 💚 mvnsite 2m 29s the patch passed
-1 ❌ javadoc 1m 10s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 1m 39s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
-1 ❌ spotbugs 1m 31s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 38m 17s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 36s hadoop-common in the patch passed.
+1 💚 unit 2m 56s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
261m 56s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1598]
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 9c9ebfecf71d 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 883bc1d
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/testReport/
Max. process+thread count 3140 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 7m 17s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 5m 45s Maven dependency ordering for branch
+1 💚 mvninstall 17m 57s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 8m 32s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 7m 46s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 1m 54s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 1m 28s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 1m 21s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 5s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 2m 13s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 22m 44s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
+1 💚 mvninstall 0m 52s the patch passed
+1 💚 compile 8m 37s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 8m 37s the patch passed
+1 💚 compile 8m 22s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 8m 22s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 54s /results-checkstyle-root.txt root: The patch generated 275 new + 23 unchanged - 0 fixed = 298 total (was 23)
+1 💚 mvnsite 1m 30s the patch passed
-1 ❌ javadoc 0m 42s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 1m 7s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
-1 ❌ spotbugs 0m 58s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚 shadedclient 20m 8s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 49s hadoop-common in the patch passed.
+1 💚 unit 2m 17s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
146m 59s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1609]
Unread field:S3ABlockOutputStream.java:[line 1603]
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 8cf9a31cfabb 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 9e5ff77
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/testReport/
Max. process+thread count 3150 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steveloughran steveloughran steveloughran requested changes

Assignees

@steveloughran steveloughran

Labels
AWS build Common DistCp MapReduce TOOLS trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@saikatroy038 @hadoop-yetus @steveloughran