[Feature]: Load Playwright settings programmatically as an alternative to a .runsettings file

[0xced]

🚀 Feature Request

Currently, setting the BrowserTypeLaunchOptions when running tests with the Microsoft.Playwright.MSTest, Microsoft.Playwright.NUnit or Microsoft.Playwright.Xunit pacakge is controlled through the .runsettings configuration file.

It would be nice to be able to control the launch options programmatically, just like BrowserNewContextOptions can be controlled programmatically by overriding the ContextOptions() method.

Example

using System;
using System.Threading.Tasks;
using Microsoft.Playwright;
using Microsoft.Playwright.TestAdapter;
using Microsoft.Playwright.Xunit;
using Xunit;

public class IntegrationTests(WebAppFixture webAppFixture) : PageTest, IClassFixture<WebAppFixture>
{
    // 👇 Proposed new virtual method where browser, launch options and more can be controlled programmatically
    protected override PlaywrightSettings PlaywrightSettings()
    {
        return new PlaywrightSettings
        {
            Browser = PlaywrightBrowser.Firefox,
            LaunchOptions = new BrowserTypeLaunchOptions
            {
                Headless = false,
                SlowMo = 100,
                Timeout = 20_000,
            },
            ExpectTimeout = TimeSpan.FromSeconds(10),
        };
    }

    [Fact]
    public async Task TestSearchPage()
    {
        await Page.GotoAsync(webAppFixture.Url("search"));
        await Expect(Page).ToHaveTitleAsync("Search");
    }
}

This example is given with Xunit but would work the same with MSTest or NUnit.

Motivation

Controlling options programatically opens up new possibilities that are not achievable with a .runsettings file. For example, one could decide to run tests on Firefox based on some runtime conditions or change the value of the Timeout based on what hardware the tests are running on. This would give a lot of power to Playwright users.

Also, having a new PlaywrightSettings type is great for discoverability. Typing properties with autocompletion support from your IDE is a much better experience than writing XML elements in a .runsettings file without any autocompletion.

Finally, .runsettings support in Rider is pretty broken, I was not able to get the Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ISettingsProvider.Load method to be called, rendring Playwright .runsettings unusable with Rider. See also TestCaseFilter in .runsettings ignored.

[mxschmitt]

We intentionally had these settings configured on a global level before - this would require us to diverge from what we do in Node.js or Python. Usually for end-to-end tests, you write a test once and run them on all the browsers. For that you run dotnet test 3 times. I recommend to reach out to Jetbrains to fix the issue with runsettings - in the CLI it seems to work all expected, so its an issue with the IDE. Do you mind elaborating why you want to run some test files only in a specific browser? We recommend having multiple runsettings files if you want to specify custom timeouts etc.

[Xen0byte]

Hi @0xced, if it helps with what you need, I usually prefer to have a JSON configuration file and define my browser options in there under different profiles, then I just deserialise as needed and pass that object to the browser on initialisation. The profile is selected by name by an environment variable of sorts, so it can be dynamically set, e.g. .runsettings, CLI command argument, or pipeline variable. Some of the reasons for which I prefer this method are that 1) it keeps the config out of the code, which is what I believe that the official .runsettings approach is aiming for, but 2) is more powerful, flexible, reusable, and transparent than .runsetting config.

[0xced]

Usually for end-to-end tests, you write a test once and run them on all the browsers. For that you run dotnet test 3 times.

That's one way to do it, I personally prefer to control how tests are run with C# code rather than in yaml or shell scripts.

Do you mind elaborating why you want to run some test files only in a specific browser?

For example, to test on older systems where the newest versions of Chromium and WebKit are not supported anymore. But again, the point is to control all of this from C# instead of yaml or shell scripts.

If #3082 is not accepted, it's still possible to load the settings programmatically without being part of Playwright test adapter project. It's just much uglier because one has to go through the hoops of creating an XmlReader and making sure to install the settings early enough to be picked by Playwright. (The private static PlaywrightSettingsXml _settings field must be set before accessing any of the PlaywrightSettingsProvider properties, which can only be guaranteed if it's integrated into the Playwright test adapter project.)

using System;
using System.Linq;
using System.Text.Json;
using System.Xml;
using System.Xml.Linq;
using Microsoft.Playwright;
using Microsoft.Playwright.TestAdapter;

namespace SampleCode;

public class PlaywrightSettings
{
    public PlaywrightBrowser Browser { get; set; } = PlaywrightBrowser.Chromium;
    public BrowserTypeLaunchOptions? LaunchOptions { get; set; }
    public TimeSpan? ExpectTimeout { get; set; }

    public void Install()
    {
        using var reader = CreateXmlReader();
        new PlaywrightSettingsProvider().Load(reader);
    }

    private XmlReader CreateXmlReader()
    {
        var playwright = new XElement("Playwright", new XElement("BrowserName", Browser));
        if (LaunchOptions != null)
        {
            playwright.Add(CreateLaunchOptions(LaunchOptions));
        }

        if (ExpectTimeout != null)
        {
            playwright.Add(new XElement(nameof(ExpectTimeout), ExpectTimeout.Value.TotalMilliseconds));
        }

        return playwright.CreateReader();
    }

    private static XElement CreateLaunchOptions(BrowserTypeLaunchOptions options)
    {
        var launchOptions = new XElement(nameof(LaunchOptions));
        if (options.Args != null) launchOptions.Add(new XElement(nameof(options.Args)), JsonSerializer.Serialize(options.Args));
        if (options.Channel != null) launchOptions.Add(new XElement(nameof(options.Channel)), options.Channel);
        if (options.ChromiumSandbox != null) launchOptions.Add(new XElement(nameof(options.ChromiumSandbox)), options.ChromiumSandbox);
#pragma warning disable CS0612 // Type or member is obsolete
        if (options.Devtools != null) launchOptions.Add(new XElement(nameof(options.Devtools)), options.Devtools);
#pragma warning restore CS0612 // Type or member is obsolete
        if (options.DownloadsPath != null) launchOptions.Add(new XElement(nameof(options.DownloadsPath)), options.DownloadsPath);
        if (options.Env != null) launchOptions.Add(new XElement(nameof(options.Env)), JsonSerializer.Serialize(options.Env.ToDictionary(x => x.Key)));
        if (options.ExecutablePath != null) launchOptions.Add(new XElement(nameof(options.ExecutablePath)), options.ExecutablePath);
        if (options.FirefoxUserPrefs != null) launchOptions.Add(new XElement(nameof(options.FirefoxUserPrefs)), JsonSerializer.Serialize(options.FirefoxUserPrefs.ToDictionary(x => x.Key)));
        if (options.HandleSIGHUP != null) launchOptions.Add(new XElement(nameof(options.HandleSIGHUP)), options.HandleSIGHUP);
        if (options.HandleSIGINT != null) launchOptions.Add(new XElement(nameof(options.HandleSIGINT)), options.HandleSIGINT);
        if (options.HandleSIGTERM != null) launchOptions.Add(new XElement(nameof(options.HandleSIGTERM)), options.HandleSIGTERM);
        if (options.Headless != null) launchOptions.Add(new XElement(nameof(options.Headless)), options.Headless);
        if (options.IgnoreAllDefaultArgs != null) launchOptions.Add(new XElement(nameof(options.IgnoreAllDefaultArgs)), options.IgnoreAllDefaultArgs);
        if (options.IgnoreDefaultArgs != null) launchOptions.Add(new XElement(nameof(options.IgnoreDefaultArgs)), options.IgnoreDefaultArgs);
        if (options.Proxy?.Server != null) launchOptions.Add(new XElement(nameof(options.Proxy)), CreateProxy(options.Proxy));
        if (options.SlowMo != null) launchOptions.Add(new XElement(nameof(options.SlowMo)), options.SlowMo);
        if (options.Timeout != null) launchOptions.Add(new XElement(nameof(options.Timeout)), options.Timeout);
        if (options.TracesDir != null) launchOptions.Add(new XElement(nameof(options.TracesDir)), options.TracesDir);
        return launchOptions;
    }

    private static XElement CreateProxy(Proxy value)
    {
        var proxy = new XElement(nameof(Proxy), new XAttribute(nameof(value.Server), value.Server));
        if (value.Bypass != null) proxy.Add(new XElement(nameof(value.Bypass), value.Bypass));
        if (value.Username != null) proxy.Add(new XElement(nameof(value.Username), value.Username));
        if (value.Password != null) proxy.Add(new XElement(nameof(value.Password), value.Password));
        return proxy;
    }
}