Azure extension for Elasticsearch install is failing

[ColeSiegelTR]

I'm using ARM templates to deploy and have got my master node VMs deployed. However when I attempt to run the elasticsearch install script, I am getting error code 10 (https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/scripts/elasticsearch-install.sh) using version 7.11.1

Any suggestions on how to troubleshoot this? I've seen a bunch of closed issues regarding this error stating that it was fixed but I am not sure if the root cause was the same.

The command executed by VM and result is below.

bash elasticsearch-install.sh -xdn 'elasticsearch' -v '7.11.1' -m 0 -A 'Abcdefg1234567!' -R 'Abcdefg1234567!' -K 'Abcdefg1234567!' -S 'Abcdefg1234567!' -F 'Abcdefg1234567!' -M 'Abcdefg1234567!' -B 'Abcdefg1234567!' -Z 3 -p 'cluster1' -L '' -C '' -D 'XX.XXX.XXX.X' -H '' -G '' -V '' -J '' -T '' -W '' -N '' -O '' -P ''

{
  "code": "DeploymentFailed",
  "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
  "details": [
    {
      "code": "VMExtensionProvisioningError",
      "message": "
VM has reported a failure when processing extension 'script'. Error message: 
\"Enable failed: failed to execute command: command terminated with exit status=10
[stdout]
[04052021-20:51:32] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 60/60
[04052021-20:51:33] [wait_for_started] never saw elasticsearch go up locally

[stderr]
  0     0    0     0    0     0 --:--:--  0:00:14 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:22 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:23 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:24 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:25 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:26 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:28 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:29 --:--:--     0
100   151  100   151    0     0      5      0  0:00:30  0:00:30 --:--:--    31
100   151  100   151    0     0      5      0  0:00:30  0:00:30 --:--:--    39
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:22 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:23 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:24 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:25 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:26 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:28 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:29 --:--:--     0
100   151  100   151    0     0      5      0  0:00:30  0:00:30 --:--:--    31
100   151  100   151    0     0      5      0  0:00:30  0:00:30 --:--:--    39
\"



More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "
    }
  ]
}

[mal-clue]

I've been hitting the same problem since last week. The following is the output in /var/logs/elasticsearch/elasticsearch.log:

[2021-05-12T09:29:15,918][DEBUG][o.e.a.ActionModule] [data-0] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2021-05-12T09:29:16,320][ERROR][o.e.b.Bootstrap] [data-0] Exception
java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.keystore.path] setting
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport.<init>(SecurityNetty4HttpServerTransport.java:51) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$19(Security.java:1092) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:515) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.14.jar:6.8.14]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.14.jar:6.8.14]
[2021-05-12T09:29:16,329][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [data-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.ke$
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.14.jar:6.8.14]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.14.jar:6.8.14]
Caused by: java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.keystore.path] setting
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport.<init>(SecurityNetty4HttpServerTransport.java:51) ~[?:?]
at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$19(Security.java:1092) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:515) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.14.jar:6.8.14]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.14.jar:6.8.14]
... 6 more

Comparing /etc/elasticsearch/elasticsearch.yml to a known good version shows the certificate information missing:

...
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true

Known good version:

...
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/ssl/elasticsearch-http.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/ssl/elasticsearch-http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/ssl/elasticsearch-transport.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/ssl/elasticsearch-transport.p12

The certificates look to exist in /etc/elasticsearch/ssl but the config isn't pointing to them. Sure, I can edit the yml manually, but the rest of the bootstrap script will not have run after the nodes came online, so I'd rather the issue was fixed. ASAP, if possible, as this is holding up some customer deployments for us.

[ColeSiegelTR]

We ultimately decided to go with AWS managed Elasticsearch offering. We were looking to spin something up rather quickly and the azure solution seems to require a great deal of customization and troubleshooting.

[mal-clue]

Having dug into this deeper, I found that the .p12 certificates hadn't actually been generated and elasticsearch-certutil was failing silently with the following exception:

Encrypt Private Key failed: unrecognized algorithm name: PBEWithSHA1AndDESede

...which relates to a new issue in Open JDK (Oracle). As the desired state for the ARM template gets the latest version of Open JDK, the issue is apparent on pretty much any version of the ARM template.

https://bugs.openjdk.java.net/browse/JDK-8266261

I'm assuming we cannot specify the Open JDK version to install when provisioning...

Can the template be updated to target a specific Open JDK version?

[mal-clue]

FWIW, I need to target Elasticsearch 6.8.x and therefore cannot use the bundled JDK (which may not exhibit the issue):

# Only install Java if not bundled with Elasticsearch
if [[ -z "$ES_VERSION" || $(dpkg --compare-versions "$ES_VERSION" "lt" "7.0.0"; echo $?) -eq 0 ]]; then
  install_java
else
  log "not installing java, using JDK bundled with distribution"
fi

[russcam]

Thanks for opening @ColeSiegelTR, and for the additional details and investigation @mal-clue. My apologies that it has taken some time to respond.

We'll need to investigate to see what we can do to mitigate this.

[mal-clue]

Thanks @russcam - this is still very much an issue for us and we are having to work around our deployments.

[mal-clue]

@russcam is there any progress on this? This is very much still a huge issue for us. Thanks.

[russcam]

@mal-clue I no longer work on this project; I'll see if there is someone who can take a look at this.

[mikepetridisz]

@mal-clue could you figure out something? We are hitting this error as well.

[mal-clue]

@mikepetridisz Our horrible workaround involves deploying without using HTTPS for the transport (9300) or HTTP (9200). That means that elasticsearch-certutil isn't involved as it isn't required and that is what is ultimately causing the issue.

We're also investigating whether we can perform a set of post-deploy steps using Azure CLI to:

• install OpenJDK9
• enable the security on the Transport Layer (generating a CA using elasticsearch-certutil)
• enable the security on the HTTP Layer (generating the node certificates using elasticsearch-certutil)

elasticsearch-certutil appears to work fine when running under OpenJDK9.

Essentially, we're hacking our way around this. We could possibly fork the repo and update the scripts ourselves although I'm not sure how much effort is involved in that.

@russcam I'm a little concerned over the lack of support for this - is this on Elasticsearch's radar? Is this project now dead?