Add test - delete an engine with a running build

[johnml1135]

It gave a few errors - it should close down gracefully.

Specifically, if a new engine is created and is in the process of being built (pending or active), the jobs should cancel before it is all deleted. There may also be a need for a state "deleting" ... maybe...

[johnml1135]

The abbreviated error message showing the need:

Call failed with gRPC error status. Status code: 'NotFound', Message: 'The build does not exist.'.\n","stream":"stdout","time":"2023-09-11T16:18:29.255840722Z","type":"info","source":"Grpc.Net.Client.Internal.GrpcCall"}
POST http://serval-api:81/serval.translation.v1.TranslationPlatformApi/BuildFaulted\n","stream":"stdout","time":"2023-09-11T16:18:29.215850556Z","type":"info","source":"System.Net.Http.HttpClient.TranslationPlatformApiClient.LogicalHandler"}
POST http://machine-engine/serval.translation.v1.TranslationEngineApi/Delete\n","stream":"stdout","time":"2023-09-11T16:18:27.478076473Z","type":"info","source":"System.Net.Http.HttpClient.Nmt.ClientHandler"}
POST http://serval-api:81/serval.translation.v1.TranslationPlatformApi/BuildStarted\n","stream":"stdout","time":"2023-09-11T16:17:51.818285896Z","type":"info","source":"System.Net.Http.HttpClient.TranslationPlatformApiClient.ClientHandler"}

[johnml1135]

may be related to #121 - about things happening that aren't quite complete.

[johnml1135]

Or, we disallow deletion on an engine if there is an active build and tell the user to cancel the build first.

[ddaspit]

This sounds like a bug. It should cancel the build automatically.

[johnml1135]

So, will we also need a "deleting" state? Or is that superfluous? Once it is deleted, there is no expectation of it existing. Yet, the cancelling and deleting may take 20 seconds to fully push through the whole system. How should this be handled?

[ddaspit]

We don't need a "deleting" state for builds. The "canceling" state should suffice. Right now, the delete request blocks until the current build has stopped and the engine was successfully deleted. I don't love this, but the alternative is to make the delete operation asynchronous, which means that we don't have a good way to report a failed delete operation back the calling application.

[johnml1135]

If that is the current behavior - then let's make sure that it is tested fully and I will be ok. We should be able to do this with unit tests at the Machine level. @ddaspit do you agree?

[ddaspit]

Yeah, sounds good.

[johnml1135]

To implement: Nmt and Smt test - delete the engine while the engine is building. Unit tests in Machine. May end up just being implemented in sillsdev/machine#101.

[Enkidu93]

Covered here and here.