Support renaming files on Transfer Manager Upload

[sydney-munro]

No description provided.

[chalmagr]

Having this same issue... adding some more details

Environment details

OS type and version: MacOS 14.6.1 / Windows 11 (issue happens on both)
Java version: Temurin-11.0.19+7 (build 11.0.19+7)
Version(s): 2.42.0

Steps to reproduce

1. Implement the shared code in the documentation to upload multiple files (https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-java)
2. Run the code trying to upload one directory (e.g. /Users/me/source/dir) to some location in a bucket: e.g. gs://my-bucket-name/destination (directory contents are file1.txt, file2.txt) (run the code with arguments

Code example

package org.example;

import com.google.cloud.storage.transfermanager.ParallelUploadConfig;
import com.google.cloud.storage.transfermanager.TransferManager;
import com.google.cloud.storage.transfermanager.TransferManagerConfig;
import com.google.cloud.storage.transfermanager.UploadResult;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;

public class Main {
    public static void main(String[] args) throws IOException {
        uploadDirectoryContents(args[0], Paths.get(args[1]), args[2]);
    }

    private static void uploadDirectoryContents(String bucketName, Path sourceDirectory, String remotePath) throws IOException {
        TransferManager transferManager = TransferManagerConfig.newBuilder().build().getService();
        ParallelUploadConfig parallelUploadConfig = ParallelUploadConfig.newBuilder()
                .setBucketName(bucketName)
                .setPrefix(remotePath)
                .build();

        List<Path> filePaths = new ArrayList<>();
        try (Stream<Path> pathStream = Files.walk(sourceDirectory)) {
            pathStream.filter(Files::isRegularFile).forEach(filePaths::add);
        }
        List<UploadResult> results = transferManager.uploadFiles(filePaths, parallelUploadConfig).getUploadResults();

        for (UploadResult result : results) {
            System.out.println(
                    "Upload for "
                            + result.getInput().getName()
                            + " completed with status "
                            + result.getStatus());
        }
    }
}

Run the above example using arguments
my-bucket-name /Users/me/source/dir destination

I would expect that the bucket contains the following files now

gs://my-bucket-name/destination/file1.txt
gs://my-bucket-name/destination/file2.txt

However, the bucket contains the following (for a MacOS scenario)

gs://my-bucket-name/destination/Users/me/source/dir/file1.txt
gs://my-bucket-name/destination/Users/me/source/dir/file2.txt

For Windows, the /Users/me/source/dir/file1.txt gets changed to C:\Users\me\source\dir\file1.txt

[chrisrhut]

Hi Google Cloud team, is there any way to vote or bump this up?

As it currently stands the TransferManager cannot feasibly be used in an application server context because we would have to mimic the entire object tree on our server's file system.

For example, if we want objects to end up in BUCKET_NAME/contents/upload/photo.jpg then we would have to have the /contents directory available on the server's FS. (Or, if using a prefix, /upload.) This isn't really scalable since it completely binds the app logic to the server configuration (ensuring the needed root dirs exist etc).

A system to logically map local filesystem paths to their destinations in GCS would neatly solve this issue.

(I'm happy to try a PR if nobody else is working on this.)

[BenWhitehead]

We're currently scoping this and hope to have an implementation fairly soon.