Shards intermittently failing when snapshotting to S3

[BrandesEric]

Hey guys, we've been using the AWS ES plugin for some time with no issues. In the past few months we've started seeing random shard failures when doing snapshots. The problem appears to be getting worse. Here is the error we see:

{"node_id":"JIXdRZ6pRBqbuSKzz4TlXA","index":"errors","reason":"IndexShardSnapshotFailedException[[errors][8] Failed to perform snapshot (index files)]; nested: IOException[Unable to upload object Prod/indices/errors/8/__7i]; nested: AmazonClientException[Unable to execute HTTP request: our_bucket.s3.amazonaws.com]; nested: UnknownHostException[our_bucket.s3.amazonaws.com]; ","shard_id":8,"status":"INTERNAL_SERVER_ERROR"}

We're snapshotting ~70 shards and maybe 4-10 experience this issue during a snapshot. Any ideas what's going on?

[BrandesEric]

I've been investigating further, and no matter what I do, it always tries to use bucket.s3.amazon.com instead of s3.amazon.com/bucket. Is there any way for me to specify the latter url structure? GIven this issue seems DNS related, I'm grasping at straws but want to try the non-vanity domain name.

[dadoonet]

I guess you could use "cloud.aws.s3.endpoint"?

Would this work for you?

[BrandesEric]

I tried that - and it was still prefixing the bucket name, as a subdomain,
rather than appending it as a url segment.

On Sat, Mar 7, 2015 at 1:36 PM, David Pilato [email protected]
wrote:

I guess you could use "cloud.aws.s3.endpoint"?

Would this work for you?

—
Reply to this email directly or view it on GitHub
#189 (comment)
.

[dadoonet]

I see. In that case bucket name should may be not mandatory So you can keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as s3 and base_path with your bucket name?

[BrandesEric]

Now we're talking :) I'm trying one additional DNS tweak but if that
doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato [email protected]
wrote:

I see. In that case bucket name should may be not mandatory So you can set
keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as s3
and base_path with your bucket name?

—
Reply to this email directly or view it on GitHub
#189 (comment)
.

[BrandesEric]

{

"error": "RemoteTransportException[[Warpath][inet[/10.4.0.5:9300]][cluster:admin/repository/put]];

nested: RepositoryException[[Prod_2015-03-08T00:36:17.4724345+00:00]
failed to create repository]; nested: CreationException[Guice creation
errors:\n\n1) Error injecting constructor,
java.lang.IllegalArgumentException: Bucket name should be between 3
and 63 characters long\n at
org.elasticsearch.repositories.s3.S3Repository.()\n at
org.elasticsearch.repositories.s3.S3Repository\n at
Key[type=org.elasticsearch.repositories.Repository,
annotation=[none]]\n\n1 error]; nested:
IllegalArgumentException[Bucket name should be between 3 and 63
characters long]; ",

"status": 500

}

On Sat, Mar 7, 2015 at 2:06 PM, Eric Brandes [email protected] wrote:

Now we're talking :) I'm trying one additional DNS tweak but if that
doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato [email protected]
wrote:

I see. In that case bucket name should may be not mandatory So you can
set keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as
s3 and base_path with your bucket name?

—
Reply to this email directly or view it on GitHub
#189 (comment)
.

[BrandesEric]

Oops, send got clicked before I was done :)

I'm going to try a different region, see if I can get something longer than
3 chars.

On Sat, Mar 7, 2015 at 6:38 PM, Eric Brandes [email protected] wrote:

{

"error": "RemoteTransportException[[Warpath][inet[/10.4.0.5:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[Prod_2015-03-08T00:36:17.4724345+00:00] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.IllegalArgumentException: Bucket name should be between 3 and 63 characters long\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: IllegalArgumentException[Bucket name should be between 3 and 63 characters long]; ",

"status": 500

}

On Sat, Mar 7, 2015 at 2:06 PM, Eric Brandes [email protected]
wrote:

Now we're talking :) I'm trying one additional DNS tweak but if that
doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato [email protected]
wrote:

I see. In that case bucket name should may be not mandatory So you can
set keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as
s3 and base_path with your bucket name?

—
Reply to this email directly or view it on GitHub
#189 (comment)
.

[BrandesEric]

I give up, nothing I can set seems to change the url to the configuration I
want. Ah well, have to figure something else out.

On Sat, Mar 7, 2015 at 6:41 PM, Eric Brandes [email protected] wrote:

Oops, send got clicked before I was done :)

I'm going to try a different region, see if I can get something longer
than 3 chars.

On Sat, Mar 7, 2015 at 6:38 PM, Eric Brandes [email protected]
wrote:

{

"error": "RemoteTransportException[[Warpath][inet[/10.4.0.5:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[Prod_2015-03-08T00:36:17.4724345+00:00] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.IllegalArgumentException: Bucket name should be between 3 and 63 characters long\n  at org.elasticsearch.repositories.s3.S3Repository.<init>()\n  at org.elasticsearch.repositories.s3.S3Repository\n  at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: IllegalArgumentException[Bucket name should be between 3 and 63 characters long]; ",

"status": 500

}

On Sat, Mar 7, 2015 at 2:06 PM, Eric Brandes [email protected]
wrote:

Now we're talking :) I'm trying one additional DNS tweak but if that
doesn't work I'll give this a try and let you know how it goes!

On Sat, Mar 7, 2015 at 1:49 PM, David Pilato [email protected]
wrote:

I see. In that case bucket name should may be not mandatory So you can
set keep it as null but add base_path as the one you expect?

A workaround could be to set endpoint as amazon.com and bucket name as
s3 and base_path with your bucket name?

—
Reply to this email directly or view it on GitHub
#189 (comment)
.