Testing

[shroffk]

@claudio-rosati @willrogers @kasemir @mattclarke

Now that we have a few apps, a set of framework modules, and are about to start with the data connection layer...it might be a good time to think about and put in place the testing framework for phoebus.

Here are somethings I feel we should address

• UI testing
• Utility test methods for testing implementation of framework api's (make it easier for users to write tests for implementation of framework interfaces)
• Integration testing / using mock objects (especially for services)

[claudio-rosati]

A part the TestFX framework, for my GitHub/KNOBS framework I'm using also assertj. Alternatively/contemporarily jmock could be used too

[shroffk]

@claudio-rosati can you help me create a few example test cases in the phoebus framwork? what would we need to accomplish this.

[claudio-rosati]

I can do that when we meet at ITER.
Saturday I'll come back from vacation and I'll spend the following 2 weeks preparing the 2 papers and posters for ICALEPS.
I hope at ITER to have time to understand Phoebus and then starting some work with you. I need it to start 2 different applications (real applications, not modules of the new CS-Studio) as soon as possible.

[shroffk]

Saturday I'll come back from vacation

Hope you enjoy the last 2 days of vacation.

real applications, not modules of the new CS-Studio

Can these 2 applications be developed on phoebus/KAIROS?

[claudio-rosati]

Looking at your comments I think something is going wrong: it seems to me you're developing the new cs-studio as part of the same github project of the framework. If this is the case, the framework we'll be very difficult to be used in other projects. Am I wrong?

Moreover it seems to me you are using the word "application" to identify something called plug-in/feature in eclipse or plug-in/suite in NetBeans. For people outside CS-Studio world, "application" is the double-clickable stand-alone product one can buy at the store. Using "application" to indicate something else can contribute to confusion and, at last, to have the framework used elwhere. Am I wrong again?

[willrogers]

What is Colleferro?

[willrogers]

It should in principle be simple to extract the plugins from the framework, shouldn't it?

[claudio-rosati]

@willrogers I hate my iPhone helping me...
Instead of "called" (probably mistakenly typed) it substituted it with Colleferro, the name of a town close to Rome where I lived in the past 😜

[kasemir]

As for the repository, yes, point 1 on the roadmap indicates that we start with one repo.
That's helpful as we for example rename classes across the complete code, which can then be in one commit instead of having to commit/pull from N repos.
The plan is to split this later on.

As for naming, one "Application" is "Probe" or "PV Tree" or "PV Table" or "Scan Monitor" or "Display Builder Runtime" or ...
They could be packaged in a product with just that one application. If want to pay for that, we could offer a paypal account, but we'll also offer the source code for free.
In addition, just as in RCP, we want to be able to package multiple applications into one product.

So "Product" made from "Applications" which are made from a bunch of "Modules".
How do you suggest to rename that?

[claudio-rosati]

I would rename

• product to application,
• application to feature (I prefer the Eclipse name than the NetBeans one),
• if modules are intended as java 9 modules, then that name is fine

[shroffk]

@claudio-rosati at the last cs-studio meeting we mentioned that the single repo was just a temporary convenience which will have to be restructured in the fureture.... I am fine with either
A "product" made of "apps" which each internally consist of many "modules"
or
A "Applications" made of "features" which consist of modules"
I feel this discussion can be moved to another issue ticket. #39

Could we try to create your features (which will move to their own repo soon) here using the UI testing frameworks you described? This would be a good test of what have been created so far.

[claudio-rosati]

I need to create two different applications:

• The first one is HYDRA. it will replace the current Medusa and KNOBS. The Idea is to have a framework for gauges/meters/knobs supported by a desktop application able to graphically design a gauge/meter/knob by D&D of elemental pieces (bars, ticks, labels, etc.).
• The second one is a specific desktop IDE application for OpenXAL, based on the ideas presented at IPAC '16. The main feature(s) of this application could migrate into the new CS-Studio.

[es-sns121]

I looked into using an automated GUI testing platform called Squish for testing the JavaFX views, however their software does not work with anything greater than Java 8 (1.8). I've gotten a simple test working on a simple JavaFX view using TestFX. I plan to investigate that further.

[claudio-rosati]

The main problem with Squish is that it's not free.

On my current JavaFX development (outside CS-Studio) I'm using TestFX: it allows for JUnit based test on the UI side. I've still problems on using it on Jenkins, but I have to investigate more about Monocle that seems working well on my Mac but not with Jenkins.

I like this kind of test because is a JUnit one, allowing me to test single components. Squish, on the other hand, will allow to put under test OPIs and the system as a whole, applying tests that are more close to the user.

[es-sns121]

I've got a TestFX unit test for the LogbookUI ListSelectionDialog working on Jenkings. I also had trouble with monocle. I had to fire the buttons I wanted to test manually instead of using the TestFX "clickOn(...)" method. However this method of manually firing events does seem to work. Here is a link to the test file.

[kasemir]

My brief understanding of Monocle is that it's an implementation of the 'glass' windowing component, i.e. it's replacing the guts of JFX. That means there are different versions for JFX 8 and 9. In Evan's tests, Travis was fine running the headless test with monocle on oracle java 9, but not 10, because that would require a version of monocle for J10 - which doesn't exist?! Likewise, going to J11 and on promises to cause more monocle headaches.

Turns out the main reason for monocle is to simulate mouse movement and clicks. When you want to trigger a button by moving the mouse on top and clicking, you need monocle. For invoking Button.fire(), you don't need it. Similarly, you might need it to simulate mouse movement and click to select a list item, but you can also call ListView.getSelectionModel().select(..). For testing the fundamentals of JFX widgets, you indeed need to simulate mouse movement, but we don't want to debug a Button or ListView. We want to check if our code reacts properly to button.fire() and list selection changes, so I think we're fine without monocle.

[claudio-rosati]

I partially agree (or, I've partially understood).

The problem with TextFX is that it opens and shows a stage, and that is not good for machines that don't have a screen card (like most of out Jenkins VM slaves).

If I work on a graphic framework or on a special components (like meters, gauges, etc.) I need to be sure that something on screen is drawn as expected, and some test can be done only in the JFX Platform thread (e.g. I create a filter that filters out elements in a list, then I need to check what the ListView is displaying.

From my understanding, Monocle allowed me this thing without asking my IT or Integration people to setup a virtual screen on the VM. But if I am wrong, a you suggest that the virtual screen is the solution that all of us must adopt, I'm fine, mostly if we share also how to do it.

[kasemir]

#259 has a first example for UI testing.
It adds testfx-core and testfx-junit dependencies.
The tests can be run from the IDE or from maven on a computer with display, in which case you'll briefly see the UI under test, how it selects items in the ListView etc.
For Travis, xvfb is started and several headless options are set to run the test without physical display.

According to https://github.com/TestFX/TestFX/wiki/Travis-CI, a virtual display like xvfb is needed anyway, even with monocle. xvfb is deprecated in favor of running Xorg with a dummy display. I tried that according to http://xpra.org/trac/wiki/Xdummy, running an Xorg with Xdummy as DISPLAY=:10, and the tests work fine with that as well.

All that we get from adding monocle is that the travis built fails under Java 10, since there's no monocle for JavaFX 10. Since living without monocle is easier, I suggest to do that.
I feel that checking if UI is exactly drawn as expected is asking for trouble. The UI should allow for some slop. We care if items in a list are selected, if a button is enabled. We don't test if the button is shaded with a specific gradient. Still, if you need to check pixels for a custom control, I'd use Node.snapshot() to get an image and compare that to an expected, previously taken snapshot. Still, this might require frequent updates to the test and become hard to maintain in the long run.

[claudio-rosati]

Thank you Kay. I’ll test on my other JavaFX project where I had problems with Monocle.