Replace Json API with (Kotlin?) plugin

[t-bast]

We aren't very happy with the state of the current Json API, the akka-http libraries we use are a bit hard to read and maintain (and tests are a bit lacking).

It would be interesting to rewrite the Json API as an eclair plugin, and potentially do it in Kotlin which has better support for writing APIs and handling websockets.

[tompro]

When you say in Kotlin, do you have any specific http server lib in mind (ktor, http4k, webflux ...) or do you mean on top of Java akka-http with Kotlin.

[t-bast]

We are pretty open to suggestions at the moment.

The goal is to completely remove the akka-http layer: we'd like something that can be a completely separate plugin, that only interacts with eclair-core via the Eclair interface (https://github.com/ACINQ/eclair/blob/master/eclair-core/src/main/scala/fr/acinq/eclair/Eclair.scala) and completely replaces the contents of https://github.com/ACINQ/eclair/tree/master/eclair-node/src/main/scala/fr/acinq/eclair/api.

We found the current http api code to be hard to work with and hard to test. We'd like the replacement to be easier to read, extend and test. If you have experience with that type of things, we're interested in learning about it. The interop between Kotlin and Scala is also something we haven't yet experimented a lot with, but in theory it should be easy.

[tompro]

So playing a bit with Kotlin http server interop (ktor, http4k) in an Eclair plugin shows me yes interop is possible but it requires some heavy amount of glue code between the two runtimes. So completing Scala futures on say ktor netty is not trivial and it will clutter up the plugin code (not really readable and testable at all ;) ).
Also had some issues with bytecode compatibility when running in current Eclair release (looks like a Kotlin 1.4 compile thing though so might get fixed in time).

I also played around with the current API server and it would be trivial to rework this to a more composable and easier to read implementation. My suggestion here would be to really create an Eclair specific DSL with akka-http abstracting away all the boilerplate.

I propose a multistep approach here:

1. Rework current akka-http impl. to a read and testable version
2. Extract akka-http impl. as a plugin

Later:

1. Create a small abstraction to wire Scala runtime to Kotlin coroutines. This should allow most modern Kotlin http servers to work with the Eclair API. This is going to be non trivial but one part that wakes the hacker in me ;)
2. Try to get ktor or http4k running against the abstraction. This might take longer as currently it seem the Kotlin compiler is not including some of the bytecode required at runtime. Might be related to the fat jar approach I was using.

For current API rework (if you are interested in this approach) would you like to have it as a separate plugin right away? And if so should it be put into a separate repository or still as part of the Eclair repo?

[akumaigorodski]

I'd like to add once again that whatever the change is going to be it would be valuable if plugins could reuse an API code and build their own APIs the same way base Eclair does it.

I guess this means API code better be a part of base Eclair code, at least that's the easiest option. Maybe API-enabled plugin can depend on API plugin instead but this feels a bit off for now (maybe it's not).

[t-bast]

Thanks for that early investigation! I was hoping interop between Kotlin and Scala would be more production-ready, but we may be a bit early for that. In that case your proposal early steps sound like a good start:

1. Rework current akka-http implementation to be more readable, testable and composable -> can be done while keeping everything in the eclair-node project
2. Extract this API into a plugin -> can be done afterwards, we still need to decide where these plugins will live (either a new folder in the current codebase or a new repository eclair-plugins in the acinq github organization)

As @btcontract notes, it would be nice if other plugins could extend the API by adding their own.
I think this is doable even when the API is a plugin itself, we'd just need to implement a basic plugin dependency system (or implicitly assume the API plugin is always present).
That can be done after these first two early steps IMO.

[tompro]

Sounds good. And yes an extendable library for the API will definitely be required. Also right now a plugin has dependencies into eclair-core (Kit, Eclair, ...) as well as eclair-node. Would be good to have traits for certain types directly in a thin API base lib and then publish it to maven (currently publish local is needed)