Unable to commit index to AWS OpenSearch

[caoyang1211]

I set up a domain named "mysearch" on AWS OpenSearch. Its end point is https://search-mysearch-abcd1234.us-east-1.es.amazonaws.com/ and it's set to be public accessible. I installed Norconex http collector 3.0.0 and ElasticSearch committer 5.0.0 on my local desktop and tried to index some HTML pages and commit it to my OpenSearch domain. The committer configuration is like the following:

<committer class="ElasticsearchCommitter">
<nodes>https://search-mysearch-abcd1234.us-east-1.es.amazonaws.com/</nodes>
<indexName>genetics</indexName>
<credentials>
<username>myusername</username>
<password>mypassword</password>
</credentials>
</committer>

When I started indexing, I got an error saying "ERROR ElasticsearchCommitter - Failure occured on node: "null". Check node logs." I am not sure where the log is.

To test where the problem is, I started an ElasticSearch instance on my local desktop and changed the value to http://127.0.0.1:9200 in the config file, and it successfully index the HTML pages to my local ElasticSearch instance. So it proves that my code and config file are fine.

I also ran a command on my local desktop to index a JSON file and commit it to my AWS OpenSearch instance like this:
curl -H "Content-Type:application/json" -XPOST -u "username:password" "https://search-mysearch-abcd1234.us-east-1.es.amazonaws.com/_bulk" --data-binary "@mydata.json".
This also successfully index the JSON contents to my OpenSearch instance, which proves that there's no connection problem from my desktop to my OpenSearch instance.

So I suspect it's the setting in the config file that's causing the problem. Can you please advise if using the OpenSearch endpoint in the <nodes> element is incorrect. If so, what should I change it to?

Many thanks!

CY

[essiembre]

I am not sure why it is not working for you on AWS. Have you tried specifying the port to the AWS URL or removing the forward slash? To check the AWS logs for OpenSearch, have a look here: https://aws.amazon.com/blogs/big-data/viewing-amazon-opensearch-service-error-logs/ See if you get more information there.

[caoyang1211]

Removing the forward slash solved my problem. Can't believe I spent so much time and couldn't figure it out :) Thanks a lot Pascal, you saved my day!

[essiembre]

Well, it should be smarter than this. I will mark this as a feature request to support/eliminate forward slashes.