kwild,rpc: db and rpc timeout settings

[jchappelow]

WIP: looks fine, but need to test e2e.

This adds two timeouts with settings to kwild:

• app.rpc_timeout -- timeout for the duration of request handling for both gRPC (and thus http gateway) and JSON-RPC servers enforce the timeout. This is a server timeout.
• app.db_read_timeout -- timeout on read-only DB actions (actually engine, but using the read-only DB tx). This is a service timeout applied only in Call and Query methods of the user service.

For gRCP, rpc_timeout uses a unary interceptor that cancels the context given to the service method. If I missed an existing timeout feature in the grpc packages, please let me know. I feel like this should have already existed.

For JSON-RPC, I thought this was handled by http.Server.WriteTimeout, but shockingly that timeout only closes the connection without cancelling the http.Request's context. This change employs the http.TimeoutHandler in the standard library to do this safely without concurrent writes to the http.ResponseWriter.

[jchappelow]

Resolves #717 , although this resolution is not DB specific. The timeout is applied at the level of the RPC service request. The linked issue suggests that solution is preferred, and I think so too.

[jchappelow]

Added the service-level DB read timeout (db_read_timeout) that is just like to the previous read_timeout setting.

The RPC servers themselves also have a rpc_timeout setting, which is longer, applies to all methods handled by the server, and is necessary to deal with slow clients that either cannot send or receive fast enough and may be maliciously trying to consumer resources. The JSON server already had quite long defaults for these; I have just exposed a setting and ensured that the timeout also cancels the handler's context.

[jchappelow]

I was very surprised to find that the default for the db read timeout was 100ms on v0.6.x. Isn't that extremely short? Shortest I would consider is probably 500ms-2s

Also note that the server timeout is separate, much longer, applies to all requests, and is primarily a defensive mechanism for the server.

[jchappelow]

Regarding the gRPC request timeout, it really does look like you have to make a unary interceptor like this for per-request timeout: grpc/grpc-go#5059
@Yaiba ^

[Yaiba]

Regarding the gRPC request timeout, it really does look like you have to make a unary interceptor like this for per-request timeout: grpc/grpc-go#5059 @Yaiba ^

yeah, now I see what you mean