add elasticsearch hosting

[Alirexaa]

• Add Elasticsearch hosting resource
  close: Add Elasticsearch component #4418

Microsoft Reviewers: Open in CodeFlow

• Aspire.Hosting.Elasticsearch
• Aspire.Elastic.Clients.Elasticsearch

[Alirexaa]

@dotnet-policy-service agree

[mitchdenny]

@Alirexaa could you please create a GitHub issue over on https://github.com/dotnet/docs-aspire with a article request and link it back to this PR. We want to make sure that we are bringing up the docs along with the feature changes like this one to help make your contribution more discoverable.

[mitchdenny]

So I think on the Aspire.Hosting.Elasticsearch side things are looking pretty good (open question about username?). Just need to make sure that the XML doc comments are all there.

The component side needs more work (as @davidfowl alluded to). We will probably want an Aspire.Elasticsearch...blah package for the non deprecated package. It looks like it supports OTEL so it will be good to verify that all plumbs up nicely to the dashboard.

Overall, it is looking pretty good.

[radical]

This Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyComponentWorks(resourceName: elasticsearch) [FAIL] is getting timed out waiting for a response.

The httpclient retry log messages show:

{
   "detail" : "Unsuccessful () low level call on GET: /
	 Exception: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.

	# Audit trail of this API call:
	 - [1] BadRequest: Node: http://elastic:redacted@localhost:44039/ Took: 00:01:00.0047488
	 - [2] MaxTimeoutReached:
	# OriginalException: System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
	 ---> System.TimeoutException: The operation was canceled.
	 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
	 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
	   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   --- End of inner exception stack trace ---
	   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
	   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
	   --- End of inner exception stack trace ---
	   --- End of inner exception stack trace ---
	   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
	   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
	   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
	# Request:
	<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
	# Response:
	<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
	",
   "status" : 500,
   "title" : "An error occurred while processing your request.",
   "type" : "https://tools.ietf.org/html/rfc9110#section-15.6.1"
}

The container log dumped at the end shows:

 [container-list] Executing - docker container list --all --filter name=elasticsearch --format {{.Names}} 
 [container-list] elasticsearch-bcpfvshn-3ca60341
 [elasticsearch] Executing - docker container logs elasticsearch-bcpfvshn-3ca60341 -n 50 
 [elasticsearch] Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
 [elasticsearch] 
 [elasticsearch] ERROR: Elasticsearch died while starting up, with exit code 137

Another thing to note is that this test ran to the timeout for all of the tests - 15mins. So maybe we need to bump that. But that is if the container failure is irrelevant.

[Alirexaa]

This Aspire.EndToEnd.Tests.IntegrationServicesTests.VerifyComponentWorks(resourceName: elasticsearch) [FAIL] is getting timed out waiting for a response.

The httpclient retry log messages show:

{
   "detail" : "Unsuccessful () low level call on GET: /
	 Exception: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.

	# Audit trail of this API call:
	 - [1] BadRequest: Node: http://elastic:redacted@localhost:44039/ Took: 00:01:00.0047488
	 - [2] MaxTimeoutReached:
	# OriginalException: System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
	 ---> System.TimeoutException: The operation was canceled.
	 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
	 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
	   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   --- End of inner exception stack trace ---
	   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
	   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
	   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
	   --- End of inner exception stack trace ---
	   --- End of inner exception stack trace ---
	   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
	   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
	   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, RequestData requestData, CancellationToken cancellationToken)
	# Request:
	<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
	# Response:
	<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
	",
   "status" : 500,
   "title" : "An error occurred while processing your request.",
   "type" : "https://tools.ietf.org/html/rfc9110#section-15.6.1"
}

The container log dumped at the end shows:

 [container-list] Executing - docker container list --all --filter name=elasticsearch --format {{.Names}} 
 [container-list] elasticsearch-bcpfvshn-3ca60341
 [elasticsearch] Executing - docker container logs elasticsearch-bcpfvshn-3ca60341 -n 50 
 [elasticsearch] Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
 [elasticsearch] 
 [elasticsearch] ERROR: Elasticsearch died while starting up, with exit code 137

Another thing to note is that this test ran to the timeout for all of the tests - 15mins. So maybe we need to bump that. But that is if the container failure is irrelevant.

It seems container stopped.
I'm not sure but exit code 137 mean a process is terminated externally because of its memory consumption.
Do we have limit for this?
@radical

[eerhardt]

Now that we have "FunctionalTests" in ElasticsearchFunctionalTests, I don't think we need to add this to the EndToEnd tests.

Using the new separate Hosting tests pattern, along with functional tests there, means we don't need to have every resource/component pair in the EndToEnd tests. Instead we can just use some "core" resources and components in the EndToEnd, and the other resources can be tested in their own assembly.

cc @radical @mitchdenny @sebastienros

[radical]

Fixed the merge conflicts.

[radical]

@eerhardt Tests failing with: could not create the container {"Container": {"name":"elasticsearch-ejmsbhdf-a3b368e"}, "Reconciliation": 3, "error": "docker command 'CreateContainer' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Unable to find image 'netaspireci.azurecr.io/library/elasticsearch:8.14.0' locally\nError response from daemon: manifest for netaspireci.azurecr.io/library/elasticsearch:8.14.0 not found: manifest unknown: manifest tagged by \"8.14.0\" is not found\n'"}

[eerhardt]

@eerhardt Tests failing with: could not create the container {"Container": {"name":"elasticsearch-ejmsbhdf-a3b368e"}, "Reconciliation": 3, "error": "docker command 'CreateContainer' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Unable to find image 'netaspireci.azurecr.io/library/elasticsearch:8.14.0' locally\nError response from daemon: manifest for netaspireci.azurecr.io/library/elasticsearch:8.14.0 not found: manifest unknown: manifest tagged by \"8.14.0\" is not found\n'"}

I've pushed the image to our mirror. This should be fixed now.

[eerhardt]

FYI - I've removed the EndToEnd additions as they are no longer necessary now that we have functional tests for Elasticsearch.

[mitchdenny]

@Alirexaa thank you once again for your contribution to .NET Aspire. This PR took a while to get merged and we appreciate you hanging in there whilst we go through multiple review iterations.

Just so you know we plan on shipping this in .NET Aspire 8.1 as a preview package to allow us to gather some broader usage feedback with a few of marking the package and API as stable in .NET Aspire 8.2 when it rolls around.

We are in the process of writing the blog post for .NET Aspire 8.2, are you happy for us to give you a shout out and link to your GitHub profile in that post?

[EwoutdBoer]

I will be testing this (preview) package. @Alirexaa thanks for your work. I could now completely implement Aspire in my current project, while the only missing part was Elastic search.

[Alirexaa]

@Alirexaa thank you once again for your contribution to .NET Aspire. This PR took a while to get merged and we appreciate you hanging in there whilst we go through multiple review iterations.

Glad to working here. Thanks to you and team that gave feedback to PR.

We are in the process of writing the blog post for .NET Aspire 8.2, are you happy for us to give you a shout out and link to your GitHub profile in that post?

Yes sure.

[Alirexaa]

I will be testing this (preview) package. @Alirexaa thanks for your work. I could now completely implement Aspire in my current project, while the only missing part was Elastic search.

Glad to hear. Thanks for you feedback.