[Feature] JUnit5 integration

[yury-s]

To provide decent support of JUnit in codegen (as requested in #1039), we need to first implement basic Playwright fixtures similar to other languages. This came up in the discussion of #1366 and whether/how we can integrate https://github.com/uchagani/junit-playwright into Playwright. Below are some of the requirements we'll have for such integration and some comments regarding how it compares to the existing junit-playwright.

The execution model should be similar to the default execution model of Node.js Playwright. I.e. it should be possible to

• configure browser options per test file, so that an instance of the browser is reused by all tests in the same file
• do not parallel tests in the same file, only run in parallel tests in different classes (files), i.e. in terms of junit config:
  junit.jupiter.execution.parallel.enabled = true
  junit.jupiter.execution.parallel.mode.default = same_thread
  junit.jupiter.execution.parallel.mode.classes.default = concurrent

Builtin fixtures should create Playwright, Browser, BrowserContext and Page. Their lifetimes should match those in Node.js version, i.e.

• new context and page are created before each test
• for each test, same instance of page and context is used within the test, @beforeeach and @afterEach hooks
• page and context are closed right after last @afterEach hook finished for the test

Browser & playwright lifetimes should be bound to the thread & browser configuration. Playwright instance most likely should be thread local and can be reused by all tests running on that thread. We should try to reuse the same browser instance for running more tests that require the browser launch options. Currently, playwright internal test suite creates new browser per test class which seems to be good enough.

To keep things simple in the beginning we'd like to replace PlaywrightBrowserConfig with a configurable BrowserFactory, so that we don't have to think about config format and let the user create custom browser themselves. Eventually, we'll likely want to load the config from a json/xml/.properties file. We'll likely need to experiment with a few approaches here, to strike a balance.

[yury-s]

Looking at junit-playwright it appears that it currently closes Playwright after each test which is too aggressive, we'll need to align that with the requirements above.

[uchagani]

do not parallel tests in the same file, only run in parallel tests in different classes (files)

I don't think we can enforce this. It would be up to the end user how they configure it. I think all we can do is handle both situations (which is what junit-playwright currently does). However, we can make it so the same Playwright and Browser objects are used for all tests in the file regardless of if they are running serially or in parallel.

If I understand the requirements correctly the lifecycles should be:

Playwright - Per class
Browser - Per class
BrowserContext - Per test
Page - Per test

Is my understanding correct?

[yury-s]

I don't think we can enforce this. It would be up to the end user how they configure it.

I didn't mean enforcing it, but rather provide implementation that only supports the execution model where tests from the same class are not expected to run in parallel. If someone intentionally tries to circumvent this, they are on their own. We can throw or print a warning in some obvious cases where we detect this.

I think all we can do is handle both situations (which is what junit-playwright currently does).

We can also support second mode with parallelism at method level but I'd hold off from it for now to keep things simple.

However, we can make it so the same Playwright and Browser objects are used for all tests in the file regardless of if they are running serially or in parallel.

We cannot do this as it would mean concurrent access to the same Playwright/Browser instance from more than one thread which is not supported by Playwright java.

If I understand the requirements correctly the lifecycles should be:

Playwright - Per class Browser - Per class BrowserContext - Per test Page - Per test

Is my understanding correct?

BrowserContext - Per test Page - Per test

Yes

Playwright - Per class

It can be just thread local and reused by all tests running on that thread which can include multiple classes.

Browser - Per class

We can start with this for simplicity. Later it can be optimized to reuse the browser between more classes running on the same thread and requiring same browser configuration. E.g. if there are 3 test classes that require Chromium headless with default parameters, there is no need to close the browser between them. But if say one of the 3 classes contains tests that need to run in headed mode, we'll need to close browser and launch a new one for tests in that class.

[uchagani]

@yury-s I've started working on this and I think it will be fairly easy to support same_thread and concurrent right away. I've already got a working sample.

Your idea of using ThreadLocal makes things much simpler. When running with same_thread, everything in a class is run on the same thread and serially so ThreadLocal works great for everything. When running with concurrent, we would want each test to have its own playwright/browser/etc objects so that too works great with ThreadLocal also because each test and it's lifecycle methods run on the same thread. we just close them after each test instead of in an AfterAll hook.

we'd like to replace PlaywrightBrowserConfig with a configurable BrowserFactory, so that we don't have to think about config format and let the user create custom browser themselves.

Do you have a suggestion of what this should look like?

[yury-s]

Do you have a suggestion of what this should look like?

At first glance, something like

@UseBrowserFactory(CustomBrowserFactory.class)
@UseContextFactory(CustomContextFactory.class)

public class CustomBrowserFactory implements BrowserFactory {
    @Override
    public Browser newBrowser(Playwright playwright) {
        return playwright.chromium().launch(new BrowserType.LaunchOptions().setChannel("msedge"));
    }
}

public class CustomContextFactory implements ContextFactory {
    @Override
    public Browser newContext(Browser browser) {
        return browser.newContext(new Browser.NewContextOptions().setViewportSize(1600, 1200).setLocale("ar‑AR"));
    }
}

And if no annotation is specified, use default ones. The default could create just chromium and a context with default parameters. We discussed this offline and feel like it'd be better than introducing another option for the browser/context configs when there are existing classes for that. WDYT?

[uchagani]

I like it. Do you think it would be worth it to make 1 interface called PlaywrightFactory and have all the methods in there with just 1 annotation?

Also, I came across one downside of supporting concurrent access: BeforeAll and AfterAll hooks run in separate threads than the tests themselves so we couldn't inject a playwright fixture into the test in those hooks and then reuse the same one in the test. We could make it so an error is thrown if BeforeAll/AfterAll hooks are used with concurrent access using the playwright extension. I think this is a good compromise for now. WDYT?

[yury-s]

I like it. Do you think it would be worth it to make 1 interface called PlaywrightFactory and have all the methods in there with just 1 annotation?

I feel like it'd be easier to keep them separate but it's hard to tell without seeing it. We can toy with that and change to the one that we like more.

Also, I came across one downside of supporting concurrent access: BeforeAll and AfterAll hooks run in separate threads than the tests themselves so we couldn't inject a playwright fixture into the test in those hooks and then reuse the same one in the test. We could make it so an error is thrown if BeforeAll/AfterAll hooks are used with concurrent access using the playwright extension. I think this is a good compromise for now. WDYT?

That's a good catch. This is only a problem if you want to run each test in parallel with the others. In that case, we can take same approach as in node.js and pretend that each of the tests runs in its own JVM process and run BeforeAll/AfterAll on each thread before/after corresponding tests. Doe that make sense? This is another argument in favor not running tests from same class in parallel, in that case we don't have the problem. Behavior would still differ from node.js where we rerun beforeAll (in a new test worker process) if any of the tests in the file fail.

[uchagani]

In that case, we can take same approach as in node.js and pretend that each of the tests runs in its own JVM process and run BeforeAll/AfterAll on each thread before/after corresponding tests. Doe that make sense?

We'll need to talk about this some more because I'm not sure I understand. I'm not managing threads myself and letting JUnit handle all of that. Maybe there is a way to do this without having the BeforeAll/AfterAll repeat before each test. For now I won't worry about it.

This is another argument in favor not running tests from same class in parallel

Definitely understand. I guess I'm being a bit selfish here because I use full parallel a lot so I don't have to worry about splitting up test files just to get a faster run.

I don't think we will make the 1.38 release. I think 1.39 would be doable assuming everything goes smoothly in reviews.

[yury-s]

We'll need to talk about this some more because I'm not sure I understand. I'm not managing threads myself and letting JUnit handle all of that. Maybe there is a way to do this without having the BeforeAll/AfterAll repeat before each test. For now I won't worry about it.

Yeah, this needs more thought. I'll bring it up with the team next week, maybe someone has a better idea.

I don't think we will make the 1.38 release. I think 1.39 would be doable assuming everything goes smoothly in reviews.

No rush here, 1.39 is totally fine. It's more important we do it right and don't repeat the regrets from other language ports.

[uchagani]

@yury-s is it necessary to call BrowserContext.close() and Browser.close() or can we just call Playwright.close() when we are done with the test/test class since closing playwright will close everything else too? In the docs it says

This is similar to force quitting the browser. Therefore, you should call browserContext.close() on any BrowserContext's you explicitly created earlier with browser.newContext() before calling browser.close()

but what's the downside?

[uchagani]

@yury-s question: Currently in playwright-java, is a user allowed to create multiple browsers with the same Playwright process? For example will this work with the sync api of playwright-java?

Playwright playwright = Playwright.create();
Browser chrome = playwright.chromium.launch();
Browser firefox = playwright.firefox.launch();

// use both browsers one at a time.

Or should they close one and then launch the other?