feat: JENKINS-73837 - Adjusted DirScanner to allow empty directories in the TarArchiver #9836

DerSimeon · 2024-10-06T22:33:23Z

See JENKINS-73837 .

Testing done

Proposed changelog entries

Update the DirScanner to include empty Directories in the TarArchiver

Proposed upgrade guidelines

N/A

Submitter checklist

Give feedback

The Jira issue, if it exists, is well-described.
The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
There is automated testing or an explanation as to why this change has no tests.
New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
For dependency updates, there are links to external changelogs and, if possible, full differentials.
For new APIs and extension points, there is a link to at least one consumer.
Options

Desired reviewers

@mention

Before the changes are marked as ready-for-merge:

Maintainer checklist

Give feedback

There are at least two (2) approvals for the pull request and no outstanding requests for change.
Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
Proper changelog labels are set so that the changelog can be generated automatically.
If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).
Options

…in the TarArchiver

welcome · 2024-10-06T22:33:26Z

Yay, your first pull request towards Jenkins core was created successfully! Thank you so much!

A contributor will provide feedback soon. Meanwhile, you can join the chats and community forums to connect with other Jenkins users, developers, and maintainers.

DerSimeon · 2024-10-07T01:17:04Z

It seems that a lot of Tests fail when presented with empty directories.

MarkEWaite · 2024-10-07T14:29:08Z

It seems that a lot of Tests fail when presented with empty directories.

Thanks for proposing the pull request. One of the expectations is that the automated tests must pass before a change is merged. We usually rely on the pull request submitter to make the necessary changes so that the existing automated tests are passing and are correctly expressing what is expected from the change in the pull request. Are you willing to make those changes?

DerSimeon · 2024-10-07T20:46:48Z

Thanks for proposing the pull request. One of the expectations is that the automated tests must pass before a change is merged. We usually rely on the pull request submitter to make the necessary changes so that the existing automated tests are passing and are correctly expressing what is expected from the change in the pull request. Are you willing to make those changes?

Yes absolutely, I am already working on the VirtualFileTests.
However some support in the DirectoryBrowserSupportTest would be appreciated as I am having some difficulty understanding what's going on there.

DerSimeon · 2024-10-07T21:51:57Z

@MarkEWaite could you take a look at my proposed fix?
While looking at the failing FilePathTest it appears as if it counts directories even if they aren't empty, causing FilePath#tar to return 4 (3 files + 1 dir).

MarkEWaite · 2024-10-07T23:02:42Z

@MarkEWaite could you take a look at my proposed fix? While looking at the failing FilePathTest it appears as if it counts directories even if they aren't empty, causing FilePath#tar to return 4 (3 files + 1 dir).

I think that test failure is another indication that your change is working as desired. Before your change, the contents of that file were:

-rw-rw-r-- 0/0            2049 2024-10-07 16:59 dir/pom.xml
-rw-rw-r-- 0/0            2049 2024-10-07 16:59 pom.xml
-rw-rw-r-- 0/0            1537 2024-10-07 16:59 small.tar

After your change, an entry is written to the tar archive for the directory dir. After your change, the contents of that file are:

-rw-rw-r-- 0/0            2049 2024-10-07 16:57 dir/pom.xml
-rw-rw-r-- 0/0            2049 2024-10-07 16:57 pom.xml
-rw-rw-r-- 0/0            1537 2024-10-07 16:57 small.tar
drwxrwxr-x 0/0               0 2024-10-07 16:57 dir/

I think it is correct to change the 3 in that test to a 4.

…ow empty folders

…tararchiver

DerSimeon · 2024-10-09T17:30:48Z

@MarkEWaite hey! I have fixed all except one test. However with that last one I am going to need some support.

github-actions · 2024-10-13T10:37:15Z

Please take a moment and address the merge conflicts of your pull request. Thanks!

DerSimeon · 2024-10-14T01:16:52Z

Also closes JENKINS-49296

DerSimeon · 2024-10-14T14:00:28Z

/reviewer @jenkinsci/core-pr-reviewers

timja · 2024-10-15T10:49:28Z

@timja could you check that failing windows test? to me it looks like a flaky test.

re-triggered

basil

The new behavior has required a number of tests in core to be updated, so it is likely additional tests will need to be updated (ATH and/or plugins). Please run tests in jenkinsci/bom and jenkinsci/acceptance-test-harness with these bits to confirm no additional tests need adjustment.

I am also asking for security review of this PR, as it changes the expected behavior of a number of tests for past security fixes. The new expected behavior looks good to me, but I wanted the security team to confirm.

basil · 2024-10-15T16:28:29Z

test/src/test/java/hudson/model/DirectoryBrowserSupportTest.java

@@ -685,7 +685,7 @@ public void symlink_outsideWorkspace_areNotAllowed() throws Exception {
            assertThat(zipPage.getWebResponse().getStatusCode(), equalTo(HttpURLConnection.HTTP_OK));

            List<String> entryNames = getListOfEntriesInDownloadedZip((UnexpectedPage) zipPage);
-            assertThat(entryNames, containsInAnyOrder(
+            assertThat(entryNames, hasItems(


As in commit 14cf0bf, please strengthen the assertion by maintaining the original containsInAnyOrder and adding any newly expected entries to the test.

Will do today!

@basil my latest commit should've fixed that. Could you double check?

basil

The code looks good to me. From my perspective, this PR is now ready for ATH/PCT testing and security review.

basil · 2024-10-15T20:06:27Z

test/src/test/java/hudson/model/DirectoryBrowserSupportTest.java

@@ -987,7 +993,7 @@ public void directSymlink_forTestingZip() throws Exception {
            assertThat(zipPage.getWebResponse().getStatusCode(), equalTo(HttpURLConnection.HTTP_OK));

            List<String> entryNames = getListOfEntriesInDownloadedZip((UnexpectedPage) zipPage);
-            assertThat(entryNames, hasSize(0));
+            assertThat(entryNames, hasSize(6));


Should add a containsInAnyOrder assertion after this to match the expected 6 entries, like the others in this file.

DerSimeon · 2024-10-15T20:07:07Z

The new behavior has required a number of tests in core to be updated, so it is likely additional tests will need to be updated (ATH and/or plugins). Please run tests in jenkinsci/bom and jenkinsci/acceptance-test-harness with these bits to confirm no additional tests need adjustment.

I am also asking for security review of this PR, as it changes the expected behavior of a number of tests for past security fixes. The new expected behavior looks good to me, but I wanted the security team to confirm.

Can you tell me how to run the ATH? do I just run mvn test in the test module?

basil · 2024-10-15T20:08:31Z

See jenkinsci/acceptance-test-harness#1748.

DerSimeon · 2024-10-15T20:36:46Z

so make a pull request and change the jenkins.version property if so, where do I get the value from?
Also once it passed all its test do I just close it and reference it here?

basil · 2024-10-15T23:54:52Z

When #9836 (comment) has been addressed and you have a passing CI build, the Checks tab will provide a section with the incremental build version number, which you can then consume in jenkinsci/bom and jenkinsci/acceptance-test-harness.

DerSimeon · 2024-10-16T08:14:13Z

Archiving the artifacts ** of a freestyle job with the shell step
mkdir -p foo bar bar/one bar/two
touch baz bar/qux bar/two/f
results in the directory foo not being archived, only the other empty dirs. This seems unexpected. Why archive subdirectories but not top-level directories?

A bunch of names and labels become (at least technically) wrong with this change, like the collectFiles method in this PR, or (download all files) and the help text of the "Archive the artifacts" post-build step on the UI. Additionally, the "Archive the artifacts" post-build step links to Ant documentation for the pattern definition, and I think Ant only supports files? So that would also be inconsistent if confirmed.

JENKINS-49296's wording suggests adding an option, while this does it unconditionally. This be considered a major change, as it will affect users relying on the current behavior for archived artifacts.

What would be your proposal? So far nobody has mentioned that, and from my tests the stated behaviour could not be verified. I will retry later when I have more time.

mistaken

daniel-beck · 2024-10-16T08:52:41Z

Thanks for challenging me on this, I checked again and I am unable to reproduce the behavior 🤦 No clue what I did, or thought I saw.

daniel-beck · 2024-10-16T09:02:34Z

However, I also don't see the intended behavior for JENKINS-49296, at least for the build-in "Archive the artifacts" freestyle post-build step: Downloading artifacts does not result in archives containing empty directories. Looking on the controller file system, empty directories are also not being archived there.

DerSimeon · 2024-10-16T09:23:11Z

Maybe empty directories are being additionally filtered somewhere?

daniel-beck · 2024-10-16T12:38:08Z

Maybe empty directories are being additionally filtered somewhere?

That would be my guess. I would appreciate if you could reproduce my findings independently first though, to ensure I didn't misinterpret something I was seeing again :)

If we don't have it yet (and didn't just fail for the above reason), the addition of a test going from CreateFileBuilder (or similar for dirs) to inspecting an artifact archive zip file, confirming contents, would be useful.

daniel-beck

Noting this doesn't seem to work as expected per #9836 (comment).

DerSimeon · 2024-11-11T19:00:17Z

Hey @daniel-beck I am currently on vacation so it might take a bit for me to be able to investigate. If you can point me into the right direction and tell me where I have to implement I fix, I could do it on my laptop.

daniel-beck · 2024-11-18T11:31:25Z

point me into the right direction and tell me where I have to implement I fix

No idea, sorry. Would need to step through it to see what's wrong, and I doubt I have the time for that soon.

feat: JENKINS-73837 - Adjusted DirScanner to allow empty directories …

82dbd2f

…in the TarArchiver

DerSimeon force-pushed the fix/JENKINS-73837-empty-dirs-in-tararchiver branch from dbfc3eb to fff2bae Compare October 6, 2024 23:03

fix: Fixed spotless format violations

4f3652e

DerSimeon force-pushed the fix/JENKINS-73837-empty-dirs-in-tararchiver branch from fff2bae to 4f3652e Compare October 6, 2024 23:14

fix: Fixed most VirtualFileTests

4328df6

DerSimeon and others added 5 commits October 9, 2024 01:16

feat: JENKINS-73837 - Updated the testEOFbrokenFlush Test

4cea26a

fix: JENKINS-73837 - Fixed most of the DirectoryBrowserSupportTests

20bf90d

fix: Updated VirtualFileTests and DirectoryBrowserSupportTests to all…

14c5169

…ow empty folders

Merge branch 'jenkinsci:master' into fix/JENKINS-73837-empty-dirs-in-…

6a65f48

…tararchiver

fix: JENKINS-73837 - removed unused import to fix spotbugs check

879bbda

daniel-beck mentioned this pull request Oct 13, 2024

feat(TarArchive) added support empty dir in tarArchive #9862

Closed

github-actions bot added the unresolved-merge-conflict There is a merge conflict with the target branch. label Oct 13, 2024

DerSimeon and others added 2 commits October 13, 2024 15:43

fix: Updated VirtualFile#collectFiles to fix VirtualFileTest#list

3f887f4

Merge branch 'master' into fix/JENKINS-73837-empty-dirs-in-tararchiver

139ebe8

github-actions bot removed the unresolved-merge-conflict There is a merge conflict with the target branch. label Oct 13, 2024

DerSimeon added 2 commits October 13, 2024 15:53

fix: removed duplicated test, fixed Spotless check

b78d5d6

feat: added zip test to also close JENKINS-49296

0dc6d7d

chore: removed unused imports

67713a6

comment-ops-bot bot requested a review from a team October 14, 2024 14:00

mawinter69 mentioned this pull request Oct 14, 2024

[JENKINS-73837] Add support for archiving empty directories in TarArchiver/ZipArchiver #9871

Closed

Merge branch 'master' into fix/JENKINS-73837-empty-dirs-in-tararchiver

ab6f1e7

timja and others added 4 commits October 15, 2024 16:32

Merge branch 'master' into fix/JENKINS-73837-empty-dirs-in-tararchiver

2cf54e6

Revert unnecessary changes to minimize diff

45d34c6

Strengthen assertions

14cf0bf

No need to call collectFiles on a file

52fa89e

basil added needs-ath-build Needs to run through the full acceptance-test-harness suite and removed ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback labels Oct 15, 2024

basil reviewed Oct 15, 2024

View reviewed changes

basil added the needs-security-review Awaiting review by a security team member label Oct 15, 2024

basil requested a review from a team October 15, 2024 16:26

basil reviewed Oct 15, 2024

View reviewed changes

chore: strengthen assertions in DirectoryBrowserSupportTest

62f840b

DerSimeon requested a review from basil October 15, 2024 19:22

basil reviewed Oct 15, 2024

View reviewed changes

Merge branch 'master' into fix/JENKINS-73837-empty-dirs-in-tararchiver

dc860dc

This comment was marked as outdated.

Sign in to view

daniel-beck requested changes Nov 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: JENKINS-73837 - Adjusted DirScanner to allow empty directories in the TarArchiver #9836

feat: JENKINS-73837 - Adjusted DirScanner to allow empty directories in the TarArchiver #9836

DerSimeon commented Oct 6, 2024

Submitter checklist

Maintainer checklist

welcome bot commented Oct 6, 2024

DerSimeon commented Oct 7, 2024

MarkEWaite commented Oct 7, 2024

DerSimeon commented Oct 7, 2024

DerSimeon commented Oct 7, 2024

MarkEWaite commented Oct 7, 2024

DerSimeon commented Oct 9, 2024

github-actions bot commented Oct 13, 2024

DerSimeon commented Oct 14, 2024

DerSimeon commented Oct 14, 2024

timja commented Oct 15, 2024

basil left a comment

basil Oct 15, 2024

DerSimeon Oct 15, 2024

DerSimeon Oct 15, 2024

basil left a comment

basil Oct 15, 2024

DerSimeon commented Oct 15, 2024 •

edited

Loading

basil commented Oct 15, 2024

DerSimeon commented Oct 15, 2024 •

edited

Loading

basil commented Oct 15, 2024

This comment was marked as outdated.

DerSimeon commented Oct 16, 2024

daniel-beck commented Oct 16, 2024

daniel-beck commented Oct 16, 2024 •

edited

Loading

DerSimeon commented Oct 16, 2024

daniel-beck commented Oct 16, 2024 •

edited

Loading

daniel-beck left a comment •

edited

Loading

DerSimeon commented Nov 11, 2024

daniel-beck commented Nov 18, 2024 •

edited

Loading

feat: JENKINS-73837 - Adjusted DirScanner to allow empty directories in the TarArchiver #9836

Are you sure you want to change the base?

feat: JENKINS-73837 - Adjusted DirScanner to allow empty directories in the TarArchiver #9836

Conversation

DerSimeon commented Oct 6, 2024

Testing done

Proposed changelog entries

Proposed upgrade guidelines

Submitter checklist

Desired reviewers

Maintainer checklist

welcome bot commented Oct 6, 2024

DerSimeon commented Oct 7, 2024

MarkEWaite commented Oct 7, 2024

DerSimeon commented Oct 7, 2024

DerSimeon commented Oct 7, 2024

MarkEWaite commented Oct 7, 2024

DerSimeon commented Oct 9, 2024

github-actions bot commented Oct 13, 2024

DerSimeon commented Oct 14, 2024

DerSimeon commented Oct 14, 2024

timja commented Oct 15, 2024

basil left a comment

Choose a reason for hiding this comment

basil Oct 15, 2024

Choose a reason for hiding this comment

DerSimeon Oct 15, 2024

Choose a reason for hiding this comment

DerSimeon Oct 15, 2024

Choose a reason for hiding this comment

basil left a comment

Choose a reason for hiding this comment

basil Oct 15, 2024

Choose a reason for hiding this comment

DerSimeon commented Oct 15, 2024 • edited Loading

basil commented Oct 15, 2024

DerSimeon commented Oct 15, 2024 • edited Loading

basil commented Oct 15, 2024

This comment was marked as outdated.

DerSimeon commented Oct 16, 2024

daniel-beck commented Oct 16, 2024

daniel-beck commented Oct 16, 2024 • edited Loading

DerSimeon commented Oct 16, 2024

daniel-beck commented Oct 16, 2024 • edited Loading

daniel-beck left a comment • edited Loading

Choose a reason for hiding this comment

DerSimeon commented Nov 11, 2024

daniel-beck commented Nov 18, 2024 • edited Loading

DerSimeon commented Oct 15, 2024 •

edited

Loading

DerSimeon commented Oct 15, 2024 •

edited

Loading

daniel-beck commented Oct 16, 2024 •

edited

Loading

daniel-beck commented Oct 16, 2024 •

edited

Loading

daniel-beck left a comment •

edited

Loading

daniel-beck commented Nov 18, 2024 •

edited

Loading