RFC: framework next steps

[ianchi]

This is my personal "playground" and not really ready to be merged, but I'm sharing it to prompt the discussion.

I've been toying with four different topics:
A) Upgrading framework to Angular 4 + Material 2 (and building tools to Angular-Cli)
B) creating a Directive/Component to easily integrate ubus calls in a declarative way into HTML templates, without the need of js code
C) a Dashboard, as the main page. With the intention to find a structure that can be used to define individual widgets as JSON data
D) Theming

I have made some advances in all these aspects only to the point of having just some working proof of concept. (To try it out, some instructions on the README.md). Some screenshots small / large

Anyway, @jow- / @gmarcos87 after our last conversation I've halted further coding, until we agree on the basics. The most urgent topic is the framework choice, we must settle this asap.

At the moment we have three options:

1. current Master: AngularJS and finish migration from Boostrap to Material
2. upgrade to Angular 4 + Material 2
3. user Preact+redux (and some complementary libraries?)

My opinion:
I think 1) is not worth, as those versions are already in EOL stage, and regardless of the needed rework I think it is better to change now.

Regarding 2 vs 3 I think that there aren't any clear technical advantages, and that it mostly reduces to "personal preference".
Perhaps the most "objective" measurement could be "size", where we should add some complementary libraries to Preact, as it is not a full framework and doesn't cover the same functionality.
But regarding size, I think that we should set a Target (like the 300-350kb ipk, that we already talked), and it can be achieved with any of both.

Given my "A" playground topic, I think it is obvious which my personal preference is. ;-)

Please, speak up your mind, so that after some constructive discussion we can reach an agreement, and resume development.

[nicopace]

Hi @ianchi @gmarcos87 and @jow-
I'm part of Altermundi.net, the same organization @gmarcos87 is part of.
I participated with him and @jow- in the conversation that bootstrapped this conversation :-p

I wanted to give my two cents on this, based on the approach we had on the implementation Marcos is running.

TL;DR Would vote on (3) because of the low storage footprint, that being 'without batteries included' (aka: you pick what libraries you want) makes it more lean

The features we were looking for from a framework for our mobile app were:

• very low storage footprint, in order to fit inside the router
• Single Page App, because it makes the interface more responsive and splits responsabilities very well with the 'server'
• Mobile first approach: Despite we haven't explicitly said so, as we thought about building a mobile app... the framework must provide us with the tools to develop something for a phone easily
• Installable: we wanted for the app to load fast, and to be available no matter if it is installed everywhere. At first we started creating a Cordova App (a 'native' app developed with web technologies) but then we saw that Progressive Web Apps were much leaner.
• Multilingual: because LibreMesh is supported worldwide

We came from an AngularJS/Ionic background, so it was natural for Marcos and I to try to go through that path (also, a lot of things were already sorted out)... but... we ended up realizing that that approach just couldn't scale... The storage footprint mainly was the decisive factor... I don't have the exact figures, but I think the smallest we could shrink the app was ~600kb (marcos, can you clarify this?).

So, Marcos searched for some other solution, and he came with the idea of doing it with Preact+Redux... as he had been using Redux for the Ionic app... the migration required just to work on the view side.

The result... right now with Leaflet included (a Map library that is pretty heavy) the bundle is at 250kb, and keeps going down.

Once he started developing, he saw some opportunities in generalizing the code, to make it Customizable, it is implemented in such a way that every functionality you add is added via plugins.

Understanding you come from an Angular background as us, I'm sure once you experience it we will reach to the same conclusion (or have a very interesting conversation around the topic :) ).

Regards,

[ianchi]

Hi @nicopace, I completely agree with the list of features you state. But I still fail to see why this requires switching away from Angular.
Clearly the main issue is the bundle size. As stated before, we need to set a clear goal for the final size. For this, in some old conversations with @jow- we talked of a range between 300-400kb gziped (the final ipk).

Right now, the configuration in this PR gives a bundle of about 280kb gziped (*). I know there are still features missing, but also no work on optimizing the bundle. So I assume that the final code should be even smaller than that. (There is still much room for dead code elimination, I'll do a simple try switching from Webpack to Rollup in the building system when I have some time).
From what I see in its web page, Leaflet would add about 38k gziped, so I don't see really that big of a difference in this aspect.
This still leaves room for a charting library (there are a couple less than 15kb) to add some graphics to the UI.

Anyway, I couldn't find any repo with the app you are talking about. In our chat @marcos87 pointed me at lime-ui-ng, but I couldn't find any development of the actual client side UI there. Please, could you refer me to the correct project, so I could take a look.

Regards

(*) If you build it, it's actually much bigger, but only because in this playground I was lazy and included the whole MDI svg Icons, which are more than 2000 unused icons and 210kb gziped. Using luci2-ui-base as reference, which was stripped to only the used ones, adds less than 2kb.

[ianchi]

@jow- just by chance I came along this (prophetic?/seasoned?) post of yours from 6 months ago in the Lede Forum:

Its an early alpha at best (refering to the LuCI-NG variant linked in my earlier post). I don't know how many man hours it needs to be complete. Probably many as the entire web development ecosystem suffers from severe NIH syndrome.

The very first question I got when proposing luci-ng was why I wasn't using Angular 2 instead of Angular 1, then second was why I didn't use React instead of Angular :slight_smile:

Writing a proper ui is a thankless job I'd say. The powerusers will dismiss it as toy stuff and not really provide support, the end users will argue about styling and the web developers fight about the proper framework to use :slight_smile:

Sadly, and a bit depressing, 6 months later we are still stuck in the same discussions.... :-(
Lets hope we can quickly get over this syndrome and agree on a way to start steady advancing.

[ianchi]

So, let's change approach and focus on what we do agree on, so that we can start building forward:

From our last chat I recall:

1. Use websockets instead of jsonrpc as transport for ubus calls

2. Build a kind of "connector" as an opt in addon so that legacy Luci pages can be opened inside Luci-ng

3. Define the new Luci-ng apps/views via JSON, so that it's definition is agnostic of the implementation of the framework, and can work with different or evolving UIs.

My comments on these topics:
I agree with 1), and to be able to continue development decoupled from this definition, I encapsulated all ubus related communication in UbusService, which in turns currently uses JsonrpcService as transport. In the future the transport can be changed to websockets, and most of the rest of the application can be unaffected, upgrading only UbusService to use the new transport. Modifying the backend to add this feature is beyond my expertise, so I leave it to more skilled hands.
Regarding 2) I think that this must be worked on once we have a more stable Luci-ng and so have a clearer picture of the best approach to implement such a connector.

This leaves 3), and here we should try to define the JSON interface that we will use. Last year I sent PR #24 with a RFC for this same idea. It was a very basic first approach and being almost a year old it is a bit outdated, but perhaps we can use it to trigger the discussion. Please take a look specially at the schema definition

I think an app/view needs at least to have these data defined in one or more json files:
A. ACLs (already implemented)
B. Menu (already defined, perhaps will need some changes)
C. Optional Widget to show in the dashboard
D. The view itself, which could be further divided:
D1. Some status information (non editable part)
D2. The form for editing the underlying UCI file
D3. Some way of predefining "recipes" for standard use cases, so as to help non expert users

There are a lot of possibilities here and many aspects to consider. So looking just in a general way, I think that the definitions could be split in two aspects:

1. A schema for each UCI config file, giving information on the file itself. This could be used to define the UI but also to populate the wiki, and eventually also to complement the scripts that make the integration from UCI to the corrresponding services' config file
2. The definition of the app itself, which could be leaner and more focused as much of the info on the config is separated in its own file

Please, @jow- @nicopace @marcos87 can you share your views on these topics.

[gmarcos87]

Hi @ianchi, I agree with your comments. I was watching the version of what you did in Angular2 (very significant decrease in final size, good job!) and these days I am uploading a version almost similar in Preact as a proof of concept.

In a design where the views are defined by json, the widgets are especially important. Especially for specific functionalities. I think that having a kind of repository of widgets (for reuse) can be interesting, even think that a widget can be programmed for different consumptions (terminal, web in different flavors, mobile, etc).

For UCI we could think of a default view for methods that do not have a specific schema (a simple form).

We are currently working on orange-rpc (for web sockets), but we are adding the methods of our application to UBUS. The idea is to migrate to UBUS, even more if you are planning to support WebSocket in the not too distant future. :)

[ianchi]

I integrated the concepts of #40 and #24 adding basic UCI capabilities.
It is done as a generic UCI config editor, as the basis for later refining with schemas.
We could say it is a very basic implementation of the case @gmarcos87 mentioned, of an UCI config with no defined schema.
Any section type present in the config is shown as tabs, and within every section is shown as a card with every present option as an input.
Right now only text inputs are used, but that should be dynamic based on the type of the option.
Saving functionality is still no yet implemented.
No much work has yet been done on styling.

As next step I think we should define what an UCI schema will look like. I'm elaborating on the idea of using json-schema.org as a basis for describing the config file, as it is a well known standard and has flexibility.

Note: as this is still a dev playground, with no installation in the Lede box, it displays the menu items received from the router, but that doesn't correspond to the currently available pages. They are hardcoded as follows:
Status: shows a (very elementary) dashboard
Systems: lets you issue manual ubus calls, for testing and as proof of concept for developing an UbusDirective
Network: lets you select any uci config file present in the router, generating a dynamic form (provided you have the right permissions in ACLs).

[jow-]

1. Yes, but with abstraction. It will take me a while to implement Websockets support on the backend but I don't see it as an immediate blocker since we can emulate most stuff using traditional HTTP calls.
   
   I know that we currently cannot use things like UBUS subscriptions and server-push notifications without websockets but that should not hinder us from continuing with the general application scaffholding.

2. Yes. My idea here is to build some kind of special legacy LuCI theme which removes header and footer markup but includes some special .js file instead which takes care of rendering the LuCI-NG framing around the legacy content markup. I have already started to convert legacy LuCI to ubus login sessions so that message passing between old and new LuCI becomes easier.
   
   At this stage I am skeptical about how feasible a deeper integration is, like refactoring the old Lua/HTML CBI to output JSON schemas instead as this would require a lot of message-passing with the backend to execute embedded Lua code snippets and expose their inputs/outputs via some RPC channel.

3. I definitely support that idea as long as we provide enough "hooks" and "styling hints" to hand-adjust element placement and form layout where needed. In any case this approach will strongly enforce the spearation of views and code, whcih is good.
   
   Keep in mind however that it is not always feasible to directly map UCI configurations to views as certain user-oriented workflows involve multiple files at the same time (e.g. wireless and network are commonly used and configured together as settings span both files).

So maybe we should decouple the entire schematic approach from the actual UCI configuration but instead let the schema describe the structure of an expected nested JSON object which can be provided through different means with a straight UCI values to JSON mapping just being one of several possibilities.

This way, the view or model layer can assemble an intermediate data model representing system state from various sources like different UCI files, runtime information etc. which is then rendered into a form according to the ui schema declared for the particular view.

Regarding the structure of views I support an approach which divides every page into three areas, each of them optional:

• A status area displaying runtime state, like a listing of active leases on a DHCP settings page
• A settings area where the settings form is rendered in
• A task area offering access to a variety of correponding actions, e.g. a "kick wireless client" or "perform site survey" on the wifi settings page

[ianchi]

I agree.

Regarding points 1 and 2, they should get worked in parallel, mainly outside this repo and eventually merge some changes here.
I'm adding separate issues to track them individually, and keep there any related discussion. (#43 & #44)

Point 3, needs to be worked on here. In a few days I will send a separate PR with an initial proposal, trying to integrate all the comments so far, and we continue the specific discussion over there.

So, this leaves the current PR to the initial topic of the framework choice.
While we have work to do with the schema definition, and even though it should be framework agnostic, we still need to make a decision regarding with which tool we will make the implementation, or if there won't be only one implementation.

As stated before, I don't see any need to switch away from Angular. This PR already builds an app that is about 245kb gziped, and there is plenty of room for improvement. The Angular team is actively working on solving some tree shaking problems, and so eventually I believe the app should be below the 200kb threshold.
I do propose to switch from AngularJs to Angular, as the former is EOL, and so merge this PR and continue work from here.

[ianchi]

Update: the gziped app now is only 185kb
This mainly thanks to the work of the Angular team on optimizations to allow proper tree shaking.

@gmarcos87 @nicopace I believe that the original size concerns that triggered the proposition to move out of Angular are now covered, and so there shouldn't be a reason to change framework. Please comment

[nicopace]

We could say that the size is sorted out, congrats!
I think @gmarcos87 is the one that has a word on this.

[gmarcos87]

Hello, I tried to compile the pull request but give me other results. The build now in row is 1.9mb and compressed 470.3kb. I get it by doing ng build --prod. What could be happening?
If it is my mistake, it seems fine, closed the issue of size we can continue working. Surely you have in mind some roadmap, what tasks do you think we can contribute?
At the moment I could not go much further with my example in preact, you can see it at https://github.com/gmarcos87/preact-luci and the Libremesh app is in https://github.com/libremesh/lime-app
I did not get to implement the same functions so it does not serve to compare the results obtained.
Anyway I am more interested in knowing how the project is going on and if I can be useful in any particular issue.

[ianchi]

@gmarcos87 the difference in size comes from two reasons:
the main is because I exclude in the computation the MDI svgs file.
I mentioned it some time ago in this post:

(*) If you build it, it's actually much bigger, but only because in this playground I was lazy and included the whole MDI svg Icons, which are more than 2000 unused icons and 210kb gziped. Using luci2-ui-base as reference, which was stripped to only the used ones, adds less than 2kb.

Additionally, you have to change the build command to enable proper tree shaking:

ng build --prod --vendor-chunk=false --build-optimizer=true

Sorry I didn't include it in the scripts.
The last option is the most significant (added just some days ago, and still only in RC). The last commit I pushed changes dependency to this version, please make sure that it gets installed.

[ianchi]

Regarding next steps, I think that if we all agree to go for an approach where views/apps are completely defined in json, independent of the implementation in the client, what we need to do first is to discuss a little more in detail how this json files will be, so that we came with the best approach before starting to implement it.
I started to write some ideas and examples, so as to start the discussion from somewhere.
I'll be sending it as a separate PR later this week, and would appreciate if we all add ideas over there.