Please support JWSGI

[jonassmedegaard]

Hi,

Please consider provide integration with the pseudo standard JWSGI, as that would help streamlined deployment of your tool: https://uwsgi-docs.readthedocs.io/en/latest/JWSGI.html

[jmvanel]

Hi Jonas,
could you provide some hints / precisions about

1. your use case(s)
2. what I should implement in this (non) standard
3. some test case that proves the implementation is right

[jonassmedegaard]

I recenly volunteered to help maintain a server which involves a few instances of semantic forms.
Currently, those instances are started by running (adaptations of) scala/forms_play/dist/scripts/start.sh manually after each bootup of the system. I would like to streamline to have them started automatically, and restarted automatically if they for stop reason crash or hang or in other ways start behaving unusual regarding the resources of the system, and save logfiles that gets automatically logrotated, and allow me to optionally put the daemons on standby for instances that are only infrequently used.
In other words, I would like to have those instances supervised, and I would prefer doing so with the lightweight supervising framework uWSGI.

I believe what should be implemented is documented in at the bottom of the page I pointed to already - i.e. more specifically at https://uwsgi-docs.readthedocs.io/en/latest/JWSGI.html#jwsgi-and-scala

One way to test that it works is to load it in uwsgi together with a http server plugin.
With this code (copied from above example):

object HelloWorld {
     def application(env:java.util.HashMap[String, Object]): Array[Object] = {
             val headers = new java.util.HashMap[String, Object]() {
                     put("Content-Type", "text/html")
                     put("Server", Array("uWSGI", "Unbit"))
             }
             return Array(200:java.lang.Integer, headers , "Hello World")
     }
}

...compiled and loaded like this:

sudo apt install wget scala openjdk-11-jdk-headless uwsgi-core uwsgi-plugin-jwsgi-openjdk-11
wget https://raw.githubusercontent.com/unbit/uwsgi/master/plugins/jvm/uwsgi.java
javac uwsgi.java
scalac demo.scala
uwsgi --plugins http,jvm,jwsgi --http :8080 --http-modifier1 8 --http-modifier2 0 --jvm-classpath /usr/share/java/scala-library.jar --jwsgi HelloWorld:application

...you can access the Scala code by accessing http://localhost:8080/

NB! Above uses packages in unstable Debian sid - adapt package names for other releases, e.g. openjdk-8 for the currently stable Debian stretch.

(need for custom-compiling uwsgi class is a bug in Debian)

[jmvanel]

OK, I see the problem.
As I already said to Virtual Assembly organization,
migrating to latest version of semantic_forms will help a lot , since it never freezes now, due to catching memory errors. And of course it has a lot of new features compared to the very old version used in SemApps.
Also, I want to know what solution there is in the Play! framework for supervision. I know that the Actor facilities of Scala are good at monitoring server applications.
This does not mean that I will not implement the liaison with uwsgi.

[jonassmedegaard]

Sounds great that newer releases have improved stability. You guessed correctly that the "few instances" I am talking about getting involved with indeed are the ones hosted by Virtual Assembly :-)

My raising this issue is however not specific to instability of older releases - and would not (in itself) help there: Implementing support for JWSGI would obviously be done on top of your newest work, not whatever stale code is running at some deployment somewhere. In fact, I was unaware that the instances I now help admin is particularly buggy - after all, I have only looked at it for about 24 hours so far...

My interest in support for JWSGI is far more general:

• As an admin it would make it easier to deploy this project without requiring intimate knowledge in Scala tools (e.g. that Play! framework you now mention).
• As potential future package maintainer of this project for Debian, it would ease integration with surrounding packaging infrastructure - since uWSGI has a few years of testing already, and (it seems) Play! framework is yet to be packaged.

Thanks a lot for considering!

[jonassmedegaard]

uWSGI 2.0.17.1-9, available in Debian testing (and unstable) since today, fixes a bug which simplifies testing on those platforms.

Above Scala demo can now be served via uWSGI like this:

sudo apt install scala uwsgi-plugin-jwsgi-openjdk-11
scalac demo.scala
uwsgi --plugins http,jvm,jwsgi --http :8080 --http-modifier1 8 --http-modifier2 0 --jvm-classpath /usr/share/java/scala-library.jar --jvm-classpath /usr/share/java/uwsgi.jar --jwsgi HelloWorld:application

[jmvanel]

Thanks for reminding me this issue !

It seems that there is nothing to implement in semantic_forms application, nor in Play! framework.
My understanding is that, since uWSGI can wrap any executable program, we just have to test that it works with the executable JVM program that launches semantic_forms .

If you lauch Play (and SF) the verbose way:
bin/semantic_forms_play -verbose
you see that the executable JVM program that launches semantic_forms is
play.core.server.ProdServerStart

I wonder if it would not be even simpler if uWSGI could wrap a plain shell command.

[jonassmedegaard]

Somewhere something need to return to uWSGI the 3 objects documented in https://uwsgi-docs.readthedocs.io/en/latest/JWSGI.html

[jmvanel]

OK, I did read carefully The JWSGI interface.
Indeed, there would be something to implement. It's like a API for HTTP .

Instead of the Java track , why not this track :
https://uwsgi-docs.readthedocs.io/en/latest/AttachingDaemons.html
?

[jonassmedegaard]

JWSGI is indeed an API for HTTP: Instead of each backend application fully encoding the http protocol - and then pass http between those and a front-end web server, JWSGI (and PSGI and WSGI etc.) are more lightweight protocols optimized for backend applications.

Yes, uWSGI also supports other things than its main purpose of acting as middleware between backend http-ish services and front-end http service - but my interest is in streamlining http serving and that requires use of lighter protocols at the backend, not simply wrapping full-weight protocols.

[jonassmedegaard]

I want to allow the local administrator to minimize the resource footprint of the running services. Or to spend vast resources on caching, replication, failover, etc. But whatever chosen, I want to allow the local administrator to make such choices as much as possible across the board - independently of the computing environment of each individual backend application. uWSGI provides such middleware, for the appliations that choose to offfer one of their data in one of the lightweight interfaces supported by uWSGI.

[jmvanel]

I see; unfortunately, Play! Framework is NOT among the already implemented bridges.
Also not the underlying web server of Play! Framework :
https://doc.akka.io/docs/akka-http/current/

I wonder if the uWSGI community did implement a generic HTTP client for uWSGI, enabling to wrap a web server in a generic way ?

I have to think more, and read this doc:
https://uwsgi-docs.readthedocs.io/en/latest/JVM.html

[jonassmedegaard]

uWSGI can handle http as backend protocol - see e.g. https://serverfault.com/questions/501672/use-uwsgi-as-a-proxy-server

That is not interesting for my use cases, however: That will blow up ressource use rather than limit and tighten control over resources used.

[jmvanel]

OK.
I investigated how I can implement the API.
At the moment I investigated to sneak between Play Framework and Akka Http, playing the role of Akka Http.
So the idea is to instanciate all the SF + Play Framework objects, without starting the server :) and reuse in class AkkaHttpServer , the method
def getHandlerFor(request: RequestHeader): Either[Future[Result], (RequestHeader, Handler, Application)] = {
(actually in ancestor Server class)

Looking at the stack, this function, although private, is the first one in Play that Akk HTTP calls:

private def createServerBinding(port: Int, connectionContext: ConnectionContext, secure: Boolean): Http.ServerBinding = {

[jonassmedegaard]

Sounds sensible to me to "impersonate" Akka Http (but really, don't rely on my judgement: I only skimmed the relationship between Play and Akka today after you mentioned Akka above, and I wouldn't know how to code this at all!)

[jonassmedegaard]

Perhaps look at the place where Play supports switching between Akka and Netty, and provide uWSGI as a third option there?

[jmvanel]

That's an idea too.

This issue joins the plan, that I execute little by little for years, to get rid of Play. The only dependency I have today is their router language, and Action, Request, and Result classes (2366 lines of code in SF = semantic_forms !).
And the plan is to use directly Akka HTTP, without Play.

So, if I get to know better Akka HTTP, it will help for both this issue and the plan to use directly Akka HTTP. Also , every step I make to remove Play can contribute to a (partial) JWSGI implementation. Akka HTTP has a nice native Scala language for HTTP router. And I already have several pages using this very simple API 100% semantic_forms for web pages:

trait SemanticController {
  def result(request: HTTPrequest): NodeSeq
}

But, in any, don't expect an implementation before 2 months; this is not a small issue, and I have so many tasks on SF and LOD, etc !!!

[jonassmedegaard]

Ah ok, makes less sense to play (pun intended) along with how Play integrates with http libraries if you want to get rid of Play altogether.

Sounds really neat if SF could depend only on Akka core, and either Akka HTTP or some tiny glue code for uWSGI.

Fully understand that this will take time!

[jonassmedegaard]

Do your recent refactoring e.g. at commit 9a3224c perhaps help for this issue as well?

I don't understand the code, only noticed "new class ServicesApp" in commit message which sounds related...