Skip to content

Latest commit

 

History

History
187 lines (136 loc) · 5.76 KB

README.md

File metadata and controls

187 lines (136 loc) · 5.76 KB
Raw

S3Sync

Amazon S3 Content Synchronization with .NET.

S3Sync synchronize a directory to a S3 Bucket. It meakes bucket identical to the LocalRoot (source).

Note: Remote files that are not in the LocalRoot are removed.

How to use

You can download latest version from Release Page.

Action Full.NET .NETCore 2.0 Docker
Requirement .NETFreamework 4.7 or higher .NETCore 2.0 or higher Docker
Download s3sync_netfull.zip s3sync_netcore.tar.gz guitarrapc/s3sync
Run Extract zip and run S3Sync.exe Extract zip and run dotnet S3Sync.dll docker run guitarrapc/s3sync

Configuration

You can pass parameter to S3Sync with Arguments or Environment Variable.

Arguments Environment Variable Required?
Optional?		 Description
BucketName="string" S3Sync_BucketName Required Specify S3 BucketName to sync.
LocalRoot="string" S3Sync_LocalRoot Required Specify Local File Path to Sync.
KeyPrefix="string" S3Sync_KeyPrefix Optional Specify KeyPrefix to add to localfile when Sync.
IgnoreKeyPrefix="string" S3Sync_IgnoreKeyPrefix Optional Specify KeyPrefix to ignore on S3.
ExcludeFiles="string","string" S3Sync_ExcludeFiles Optional Specify local file names you want to exclude. (use , for multiple.)
ExcludeDirectories="string","string" S3Sync_ExcludeDirectories Optional Specify local directory names you want to exclude. (use , for multiple.)
CredentialProfile="string" S3Sync_CredentialProfile Optional Specify Credential Profile name.
Silent=bool S3Sync_Silent Optional Set true when you want to supress upload progress. (Default : false)
DryRun=bool S3Sync_DryRun Optional Set true will not change s3 but see estimate plan.
Set false to execute synchronization. (Default : true)
ContentType="string" S3Sync_ContentType Optional Specify ContentType for object. (default null and will be application/octet-stream
Region="string" S3Sync_Region Optional Specify region for the bucket. (default null and will be ap-northeast-1)

Sample

You can use dotnet to run as .NETCore.

$ dotnet S3Sync.dll BucketName=your-awesome-bucket LocalRoot=/Home/User/HogeMoge ExcludeFiles=.gitignore,.gitattributes ExcludeDirectories=.git,test

No .NETCore? You can use Full.NET as a ConsoleApp.

S3Sync.exe BucketName=your-fantastic-bucket KeyPrefix=hoge LocalRoot=C:/Users/User/HomeMoge DryRun=false

Docker Support

You can run with docker.

Run with IAM Role is recommended.

docker run --rm -v <YOUR_SYNC_DIR>:/app/sync/ -e S3Sync_BucketName=<YOUR_BUCKET_NAME> S3Sync_DryRun=false guitarrapc/s3sync

Local run without IAM Role, use AWS Credentials.

$ docker run --rm -v <YOUR_SYNC_DIR>:/app/sync/ -e S3Sync_BucketName=<YOUR_BUCKET_NAME> -e AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY> -e AWS_SECRET_ACCESS_KEY=<YOUR_SECRET> S3Sync_DryRun=false guitarrapc/s3sync

Build s3sync within docker

Build S3Sync with docker-compose. This enable you not to think about .NETCore2.0 sdk installed on your host.

docker-compose -f docker-compose.ci.build.yml up

Build artifacts will be generated in following path.

S3Sync\source\S3Sync\obj\Docker\publish

Clean up build docker container resource with down.

docker-compose -f docker-compose.ci.build.yml down

Docker Image Build

Create docker image with docker-compose.

docker-compose -f docker-compose.yml build

Credential handling

Synchronization operation requires read, write and delete objects permission.

It is recommended that you use IAM Policy and Profile to handle appropriate access right.

Configure IAM Policy

Here's some sample IAM Policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1446117060000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}

If you want to restrict access to certain Bucket, then replace * with desired bucketName.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1446117060000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::PutYourBucketName"
            ]
        }
    ]
}

Configure Profile

There are several way to set profile.

If you run S3Sync on AWS Resource, you should AWS managed profile like IAM Instance Profile. If you run S3Sync on Local Environment, you can configure your machine with "aws cli" or otheres.

aws cli onfigure sample

You can create AWS Credential Profile with AWS CLI.

aws configure --profile sample

Other options?

You can create Profile with other tools.

Or you can use following method.

public static void RegisterProfile(string profileName, string accessKey, string accessSecret)
{
    var option = new CredentialProfileOptions
    {
        AccessKey = accessKey,
        SecretKey = accessSecret
    };
    new NetSDKCredentialsFile().RegisterProfile(new CredentialProfile(profileName, option));
}

License

The MIT License (MIT)