Error Creating Index Template in AWS OpenSearch Test Environment #287

kadol92 · 2024-04-12T13:17:38Z

Hello,

During the deployment of our application in the AWS OpenSearch test environment, I encountered an issue with creating an index template using our migration script. Local testing using Docker containers (image opensearchproject/opensearch:2.7.0) was successful. However, when attempting the same in the AWS environment, I receive a 400 error when trying to create the template.

Here are the details of the error from the logs:

Caused by: com.senacor.elasticsearch.evolution.core.api.MigrationException: execution of script 'FileNameInfoImpl{version=1, description='create order achiral index template', scriptName='V0001__create_order_achiral_index_template.http'}' failed
	at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executeScript(MigrationServiceImpl.java:158) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
	at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executePendingScriptsWithLock(MigrationServiceImpl.java:96) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
	at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executePendingScripts(MigrationServiceImpl.java:73) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
	at com.senacor.elasticsearch.evolution.core.ElasticsearchEvolution.migrate(ElasticsearchEvolution.java:112) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
	at com.roche.max.global.opensearch.OpenSearchMigrationConfig.lambda$flywayMigrationStrategy$0(OpenSearchMigrationConfig.java:58) ~[!/:4.0.2]
	at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:62) ~[spring-boot-autoconfigure-3.2.0.jar!/:3.2.0]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1822) ~[spring-beans-6.1.1.jar!/:6.1.1]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1771) ~[spring-beans-6.1.1.jar!/:6.1.1]
	... 25 common frames omitted
Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [https://<URL_TO_OPEN_SEARCH>.eu-central-1.es.amazonaws.com], URI [_template/order-achiral-separation], status line [HTTP/1.1 400 Bad Request]
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>

	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-8.10.4.jar!/:8.10.4]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-8.10.4.jar!/:8.10.4]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-8.10.4.jar!/:8.10.4]
	at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executeScript(MigrationServiceImpl.java:145) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
	... 32 common frames omitted

I can create the same template manually using Postman, which indicates that the issue may not be related to the configuration of our OpenSearch client, as we were able to create the migration index successfully. Could you assist in diagnosing why the migration scripts work locally but not in the AWS environment?

Thank you for your help.

The text was updated successfully, but these errors were encountered:

xtermi2 · 2024-04-12T16:37:46Z

I have no AWS OpenSearch Instance to play around. But you could log the exact HTTP request ElasticsearchEvolution is executing and the one you do by postman. I'm quite sure there is a mismatch with the HTTP Headers or something like that.

But you should add more information:

how do you authenticate? IAM oder BasicAuth?
- Maybe Compatibility to AWS OpenSearch #236 could be helpfull if you use IAM? But looks like they are running ElasticsearchEvolution on AWS, so maybe @RiVogel can help you?
AWS OpneSerachService is also on version 2.7.0?

kadol92 · 2024-04-15T14:26:35Z

Hey, @xtermi2, authentication with BasicAuth.

Yes OpenSearch on AWS has version 2.7.0. During running tests with connection to the AWS OpenSearch I'm getting the same response as in the TEST env. 400 Bad Request.

This is really strange, cuz I can execute those calls with the postman, as I described above. Any other ideas?

RiVogel · 2024-04-15T14:28:54Z

Hey @kadol92: Does each *.http file contain exactly one request or do you have multiple requests per file?

RiVogel · 2024-04-15T14:32:35Z

... and you are sure that your integration of Elasticsearch Evolution into Flyway does not modify or replace the request files at all? Therefor I agree @xtermi2, that you log the effective Elasticsearch queries which are going to the cluster.

kadol92 · 2024-04-15T14:58:00Z

Hey @RiVogel, @xtermi2,

*.http file contains one request per file

PUT _template/order-achiral-separation
Content-Type: application/json

{
  "index_patterns": [
    "order-achiral-separation-v1*"
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "_entity_type": {
        "type": "keyword",
        "index": false,
        "doc_values": true
      },
      
      ...

I've logged the details of the request, and they appear identical. I can't spot any differences. I conducted a quick test in the MigrationServiceImpl, where I switched the client to a restTemplate client and was able to successfully execute the call, just like in Postman.

Here is the change that I made:

 var restTemplate = new RestTemplate();
            HttpHeaders headers = new HttpHeaders();
            headers.setBasicAuth("user", "password");
            headers.setContentType(MediaType.APPLICATION_JSON);
            var exchange = restTemplate.exchange("https://<URL_TO_AWS_OPEN_SEARCH>/" + request.getEndpoint(), HttpMethod.PUT, new HttpEntity<>(scriptToExecute.getMigrationScriptRequest().getBody(), headers), String.class);
//            var response = restClient.performRequest(request);

            int statusCode = exchange.getStatusCode().value();
//            int statusCode = response.getStatusLine().getStatusCode();
            if (statusCode >= 200 && statusCode < 300) {
                success = true;
            } else {
                error = Optional.of(new MigrationException(String.format(
                        "execution of script '%s' failed with HTTP status %s: %s",
                        scriptToExecute.getFileNameInfo(),
                        statusCode,
                        null)));
            }

kadol92 · 2024-04-15T15:07:13Z

Btw, here is how I setup and run this migration:

@Slf4j
@Configuration
public class OpenSearchMigrationConfig {

    @Value("${open-search.host}")
    private String host;

    @Value("${open-search.scheme}")
    private String scheme;

    @Value("${open-search.username}")
    private String username;

    @Value("${open-search.password}")
    private String password;

    @Value("${open-search.migration.index}")
    private String migrationIndex;

    @Value("${open-search.migration.location}")
    private String location;

    @Bean
    public ElasticsearchEvolution elasticsearchEvolution() throws NoSuchAlgorithmException, KeyStoreException, KeyManagementException {

        var basicCredentialsProvider = new BasicCredentialsProvider();
        basicCredentialsProvider.setCredentials(new AuthScope(host, AuthScope.ANY_PORT), new UsernamePasswordCredentials(username, password));

        var sslContext = SSLContextBuilder
                .create()
                .loadTrustMaterial(null, (chain, authType) -> true)
                .build();

        return ElasticsearchEvolution.configure()
                .setHistoryIndex(migrationIndex)
                .setLocations(List.of(location))
                .load(RestClient.builder(createHttpHost(host, scheme))
                        .setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(basicCredentialsProvider)
                                .setSSLContext(sslContext)
                        )
                        .build()
                );
    }

    @Bean
    public FlywayMigrationStrategy flywayMigrationStrategy(ElasticsearchEvolution elasticsearchEvolution) {
        return flyway -> {
            flyway.migrate();

            log.info("Starting OpenSearch migration");
            var startTimeStamp = System.currentTimeMillis();
            var successfullyExecutedScripts = elasticsearchEvolution.migrate();
            log.info("OpenSearch migration executed successfully {} migration scripts in {} ms", successfullyExecutedScripts, System.currentTimeMillis() - startTimeStamp);
        };
    }

    private HttpHost createHttpHost(String host, String scheme) {
        var validHost = host;
        var schemeIdx = host.indexOf("://");
        if (schemeIdx > 0) {
            validHost = host.substring(schemeIdx + 3);
        }

        var port = -1;
        var portIdx = validHost.lastIndexOf(":");
        if (portIdx > 0) {
            try {
                port = Integer.parseInt(validHost.substring(portIdx + 1));
            } catch (NumberFormatException var7) {
                throw new IllegalArgumentException("Invalid HTTP host: " + validHost);
            }

            validHost = validHost.substring(0, portIdx);
        } else if ("https".equals(scheme)) {
            port = 443;
        }

        return new HttpHost(validHost, port, scheme);
    }
}

This is a little modified from the initial version that I started. I added custom HttpHost creation and SSLContext, cuz I wasn't sure why it's failing, so I played around a little.

kadol92 · 2024-04-16T06:29:03Z

@RiVogel @xtermi2 any ideas why this solution works but not the one from the library? What am I missing? Or maybe there is some bug?

RiVogel · 2024-04-16T11:49:29Z

@kadol92: Doesn't your response contain any response body? For me, the http files look like this:

PUT /stuff-123/_settings
Content-Type: application/json

{
  "analysis": {
    "analyzer": {
...

And in Opensearch Dashboards I need to remove the Content-Type section. But it tells me that I did something wrong in the response body.

kadol92 · 2024-04-17T06:39:23Z

@RiVogel the only response body that I get is:

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>

Nothing more

RiVogel · 2024-04-17T09:01:25Z

When you have a look into your history index - is that one created? Or are there also problems?

I ask because I needed to sign every request which is going to AWS OpenSearch, but I do not use basic authentication but other authentication mechanisms.

kadol92 · 2024-04-22T13:16:56Z

Hey @RiVogel, yes, the history index has been created properly; the problem occurs when it tries to create the template for the first index.

Currently, I've used the RestTemplate client along with Apache's client5, and this works. The downside of this solution is that I had to modify the library because it doesn't support RestTemplate.

Ideally, it would be best to use this library without any modifications, but for now, this is a workaround.

    @Bean
    @Qualifier("openSearchRestTemplate")
    public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder, CustomResponseErrorHandler errorHandler) {
        var connectionManager = new PoolingHttpClientConnectionManager();
        var httpClient = HttpClients.custom()
                .setConnectionManager(connectionManager)
                .build();
        var requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);
        requestFactory.setConnectionRequestTimeout(Duration.ofSeconds(10));
        requestFactory.setConnectTimeout(Duration.ofSeconds(10));

        return restTemplateBuilder
                .requestFactory(() -> requestFactory)
                .rootUri(scheme + "://" + host)
                .basicAuthentication(username, password)
                .errorHandler(errorHandler)
                .build();
    }

xtermi2 self-assigned this Apr 12, 2024

gerjon-eatch mentioned this issue Oct 8, 2024

Pathnames in HTTP requests must start with a slash #368

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error Creating Index Template in AWS OpenSearch Test Environment #287

Error Creating Index Template in AWS OpenSearch Test Environment #287

kadol92 commented Apr 12, 2024

xtermi2 commented Apr 12, 2024

kadol92 commented Apr 15, 2024

RiVogel commented Apr 15, 2024

RiVogel commented Apr 15, 2024

kadol92 commented Apr 15, 2024

kadol92 commented Apr 15, 2024

kadol92 commented Apr 16, 2024

RiVogel commented Apr 16, 2024

kadol92 commented Apr 17, 2024

RiVogel commented Apr 17, 2024

kadol92 commented Apr 22, 2024

Error Creating Index Template in AWS OpenSearch Test Environment #287

Error Creating Index Template in AWS OpenSearch Test Environment #287

Comments

kadol92 commented Apr 12, 2024

xtermi2 commented Apr 12, 2024

kadol92 commented Apr 15, 2024

RiVogel commented Apr 15, 2024

RiVogel commented Apr 15, 2024

kadol92 commented Apr 15, 2024

kadol92 commented Apr 15, 2024

kadol92 commented Apr 16, 2024

RiVogel commented Apr 16, 2024

kadol92 commented Apr 17, 2024

RiVogel commented Apr 17, 2024

kadol92 commented Apr 22, 2024