Skip to content

Commit

Permalink
fix: list objects trim common prefixes that match marker prefix
Browse files Browse the repository at this point in the history
This checks to see if the common prefix is before the marker and
thus would have been returned in earlier list objects request.

The error case was aws cli listing multiple entries for the same
common prefix when the listing required multiple pagination
requests.

Fixes #778
  • Loading branch information
benmcclelland committed Sep 13, 2024
1 parent 53a7abf commit bf754f4
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions backend/walk.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,13 +191,20 @@ func Walk(ctx context.Context, fileSystem fs.FS, prefix, delimiter, marker strin

// Common prefixes are a set, so should not have duplicates.
// These are abstractly a "directory", so need to include the
// delimiter at the end.
// delimiter at the end when we add to the map.
cprefNoDelim := prefix + before
cpref := prefix + before + delimiter
if cpref == marker {
pastMarker = true
return nil
}
cpmap[cpref] = struct{}{}

if marker != "" && strings.HasPrefix(marker, cprefNoDelim) {
// skip common prefixes that are before the marker
return nil
}

cpmap[cpref+delimiter] = struct{}{}
if (len(objects) + len(cpmap)) == int(max) {
newMarker = cpref
truncated = true
Expand Down

0 comments on commit bf754f4

Please sign in to comment.