Skip to content

Commit

Permalink
Fixed receipe and trimed key prefic from "/"
Browse files Browse the repository at this point in the history
  • Loading branch information
Cyril-Lagrange committed Aug 16, 2023
1 parent 559ccc5 commit 928c7fb
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 4 deletions.
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ This component provides the following configuration parameters when it is deploy

PathName: "/local/path/to/monitor/*.ext"
BucketName: "bucket-name-where-to-upload-files"
ObjectKeyPrefix: "a prefix to add to the object name in S3"
Interval: <time in second between the scans>

PathName is a path with pattern expansion as described [here](https://docs.python.org/3/library/glob.html). Some valid examples are:
Expand All @@ -71,6 +72,8 @@ PathName is a path with pattern expansion as described [here](https://docs.pytho
```

ObjectKeyPrefix allows you to put the files in a subfolder in the S3 bucket. The object name will be : s3://BucketName/ObjectKeyPrefix/orginalfilename

You need to make sure that the role
GreengrassV2TokenExchangeRole has access to the bucket listed in the configuration.

Expand Down
3 changes: 2 additions & 1 deletion recipe.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ ComponentConfiguration:
DefaultConfiguration:
PathName: "<PLACEHOLDER PATH HERE>"
BucketName: "<PLACEHOLDER BUCKET HERE>"
ObjectKeyPrefix: "<PLACEHOLDER OBJECT PREFIX HERE>"
Interval: "1"
LogLevel: "INFO"
Manifests:
Expand All @@ -21,5 +22,5 @@ Manifests:
- URI: "s3://BUCKET_NAME/COMPONENT_NAME/COMPONENT_VERSION/aws-greengrass-labs-s3-file-uploader.zip"
Unarchive: ZIP
Lifecycle:
Run: "python3 -u {artifacts:decompressedPath}/aws-greengrass-labs-s3-file-uploader/main.py \"{configuration:/PathName}\" \"{configuration:/BucketName}\" \"{configuration:/Interval}\" \"{configuration:/LogLevel}\""
Run: "python3 -u {artifacts:decompressedPath}/aws-greengrass-labs-s3-file-uploader/main.py \"{configuration:/PathName}\" \"{configuration:/BucketName}\" \"{configuration:/ObjectKeyPrefix}\" \"{configuration:/Interval}\" \"{configuration:/LogLevel}\""
Install: "pip3 install --user -r {artifacts:decompressedPath}/aws-greengrass-labs-s3-file-uploader/requirements.txt"
7 changes: 4 additions & 3 deletions src/DirectoryUploader.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ class DirectoryUploader:
def __init__(self, pathname, bucket_name, bucket_path, interval, logger:logging.Logger ,client:StreamManagerClient=None):
self.__pathname = pathname
self.__bucket_name = bucket_name
self.__bucket_path = bucket_path
self.__bucket_path = bucket_path.removeprefix("/").removesuffix("/")
self.__stream_name = bucket_name + "Stream"
self.__status_stream_name = self.__stream_name + "Status"
self.__client = client
Expand Down Expand Up @@ -128,12 +128,13 @@ async def __scan(self, under_test=False):

# Append a S3 Task definition and print the sequence number
head, tail = ntpath.split(file)

# Create folder structure in the cloud
key = self.__bucket_path+"/"+tail

# Print for logging
print("TAIL VALUE: " + tail)
print("FINAL KEY VALUE: " + key)
self.__logger.debug("TAIL VALUE: " + tail)
self.__logger.debug("FINAL KEY VALUE: " + key)

s3_export_task_definition = S3ExportTaskDefinition(input_url="file://"+file,
bucket=self.__bucket_name,
Expand Down

0 comments on commit 928c7fb

Please sign in to comment.