What new features would you like to have in TestNG?

[krmahadevan]

Have you ever wished that you had "feature x" in TestNG? Now is the time to call them out ?

Add some details of what you would like to have in TestNG in it's newer versions ?

[krmahadevan]

To get this started, TestNG should support Java Streams in it's data driven tests. This way we can now enhance our tests to have the @DataProvider annotated methods to start returning Stream<T>

[juherr]

@krmahadevan I think you should post on the mailing list if you want more answers

[krmahadevan]

@juherr Yes. Thats the plan. I will be sharing the link to this discussion on both LinkedIn and the TestNG users google forum

[ajmera25]

Tag support for the test.

Reason - In xml let say we have n number of test but we want to run subset of test matching to some condition. if we have tag mechanism then we already provided tag to the test and thus we can pass the tag from the command line without need to make any change to the xml or code. This is similar to what we can do in cucumber/karate.

[Travisnv]

After long time testng users, I find myself in a situation where testng fw just falls behind in term of making productivity with DI.

The intention with guice library is quite a "hard force" for me through various projects which turn me off.

I think we may want to make testng more friendly with DI other than guice.

[krmahadevan]

@Travisnv - If I u'stand the ask correctly, are you suggesting that TestNG should support the notion of Dependency Injection and allow the users to pick their DI framework of choice? If yes, then can you please help add more details in terms of what all you would expect. That will help us build a user story (or) multiple user stories around the dependency injection feature and start baking it into TestNG.

[krmahadevan]

@ajmera25 - I think you can perhaps take a look at the Beanshell capabilities that TestNG offers. This will help you achieve what you are asking as tag support in TestNG. Please check https://testng.org/#_beanshell_and_advanced_group_selection

[Siizy]

I am not sure if TestNg can generate a playlist for unit test while building a project just like NUnit, MsTest do. This way we have the ability to known what is going to run, run specific test from playlist on demand, save playlist and run it when needed. I understand it would require a plugin for each different IDE but this will be a great development unless we have the feature already.
Thanks

[krmahadevan]

@Siizy - Can you please help share some additional context (Screenshots or something on those lines) so that it's more clear as to what is the ask ? Also if this is something that we want from within an IDE, we can create an issue in the TestNG eclipse plugin repo. For the intellij plugin, I think we would need to create that on the JetBrains issue tracker page.

ping @missedone

[WasifIsrar]

Technically its not a feature request but a change request
If BeforeMethod is running for every test, the behaviour for every test should be different.
If it fails for first test but passes for next test, subsequent tests should not get skipped.

[krmahadevan]

@WasifIsrar - If this is a change request, then I would request you to please help file a bug with some code samples and expected output so that we can take a look at it.

[sureshguptaprf]

There should be a configurable option to treat the test run count as '1' or the data-size while using data provider.

When treating the TC count as '1', Final TC Status should be:
Fail - if atleast 1 fail
Skip - when all skip
Pass - when all pass

[krmahadevan]

@sureshguptaprf - Do you think that this ask should be confined to a specific data driven test, or if it should be something that should be controlled globally at the <test> (or) at the <suite> level ?

[sureshguptaprf]

Distributed execution capabilities i.e. splitting TCs across multiple nodes. Everyone who uses TestNG for their API Tests ends up running their tests on a single node with parallel threads. But it limits the performance when TCs are in thousands and need to be distributed across multiple nodes.

Alternatively, many create multiple suite files which are run on multiple machines in separate TestNG processes effectively making them separate runs and reported separately.

[krmahadevan]

Are you suggesting something like https://docs.gradle.com/enterprise/test-distribution/ ?

Distributed test execution would mean that there should be TestNG agents running on remote nodes and we would need to have some sort of a master/slave setup wherein tests submitted at the master node level (the master node could just be the surefire plugin JVM for e.g., which triggers test executions) would get distributed to the different nodes (which have been configured as well).

I know that in the past @cbeust has mentioned this.

ping @cbeust @juherr - Your thoughts on this ?

[juherr]

It was a project years ago but now TestNG is just a library that runs tests.
Managing nodes could be done by any external project like the one you pointed.

Tldr: not the responsability of testng

[missedone]

@sureshguptaprf i just googled that there's a commercial offering https://testgrid.io/blog/web-stories/testng-for-parallel-and-distributed-testing/

disclaim: i haven't tried it before

[sureshguptaprf]

Enriching it's Assertion APIs, making it fluent, which currently makes us look elsewhere for better Assertion libraries.

[krmahadevan]

@sureshguptaprf - With respect to assertions, I think it doesn't add a lot of value for TestNG to try and build parity with assertion specialised libraries such as AssertJ

Since assertJ is already a specialised assertions related library and provides for all the asks from assertions, it would make sense for TestNG users to just adopt it for exorbitant assertion asks.

@juherr - What do you think ?

[juherr]

Agree. The plan is more to remove the assertion api than improve it.

[prakashnarkhede]

I need an option to skip execution of @BeforeMethod against certain set of tests.
Example -
I have 10 tests, out of it, before 9 tests, i want to execute certain pre-requsite, which i will put under BeforeMethod, but say in 1 test, i dont need those pre-requsite to be executed, so want to skip execution o beforeMethod and AfterMethod for that specific test.
I know using DI, and some workaround this can be achieved, but i would like to see a very simple approach like @test(skipBeforeMethod=true) or something simillar

[prakashnarkhede]

Real-Time Analytics and Reporting Dashboard

Currently, most of us rely on extent reports, allure, or other tools in the market, You should also have agreed that testing reports is not up to the mark.
Looking for a feature that can get rid of the extra addition of a reporting library, making test automation a little more complex, and something that focuses on providing an advanced, real-time analytics and reporting dashboard within TestNG. It aims to offer detailed insights into test execution, performance metrics, and trend analysis, facilitating better decision-making and test management.

[krmahadevan]

@prakashnarkhede - This is currently already possible in TestNG.

Here are three options (Feel free to pick whichever works best for you)

For both these options, we are going to use a custom annotation that looks like below

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

//Marker annotation
@Target(java.lang.annotation.ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface SkipConfiguration {}

Option 1 - Using Native Dependency Injection and Configuration method

Test class

import org.testng.SkipException;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.lang.reflect.Method;

public class SampleTestClass {

    @BeforeMethod
    public void beforeMethod(Method method) {
        boolean shouldSkip = method.isAnnotationPresent(SkipConfiguration.class);
        String prefix = "Running configuration setup for ";
        if (shouldSkip) {
            prefix = "Skipping configuration setup for ";
        }
        System.err.println(prefix + method.getName());
        if (shouldSkip) {
            throw new SkipException(prefix);
        }
    }

    @Test
    public void testMethod1() {}

    @Test
    @SkipConfiguration
    public void testMethod2() {}
}

Option 2 - Using TestNG Listeners

Test class

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(SimpleListener.class)
public class AnotherSampleTestClass {

    @Test
    public void testMethod1() {}

    @Test
    @SkipConfiguration
    public void testMethod2() {}
}

Listener being used

import org.testng.*;

import java.lang.reflect.Method;
public class SimpleListener implements IInvokedMethodListener {

    @Override
    public void beforeInvocation(IInvokedMethod im, ITestResult tr) {
        ITestNGMethod tm = im.getTestMethod();
        Method method = tm.getConstructorOrMethod().getMethod();
        boolean shouldSkip = method.isAnnotationPresent(SkipConfiguration.class);
        String prefix = "Running configuration setup for ";
        if (shouldSkip) {
            prefix = "Skipping configuration setup for ";
        }
        System.err.println(prefix + method.getName());
        if (shouldSkip) {
            tr.setThrowable(new SkipException(prefix));
            tr.setStatus(ITestResult.SKIP);
        }
    }
}

Option 3 - Using IHookable

Base class

import org.testng.*;

import java.lang.reflect.Method;

public class AbstractTestClass implements IHookable {
    @Override
    public void run(IHookCallBack callBack, ITestResult tr) {
        ITestNGMethod tm = tr.getMethod();
        Method method = tm.getConstructorOrMethod().getMethod();
        boolean shouldSkip = method.isAnnotationPresent(SkipConfiguration.class);
        String prefix = "Running configuration setup for ";
        if (shouldSkip) {
            prefix = "Skipping configuration setup for ";
        }
        System.err.println(prefix + method.getName());
        if (shouldSkip) {
            throw new SkipException(prefix);
        }
        callBack.runTestMethod(tr);
    }
}

Test class

import org.testng.annotations.Test;

public class YetAnotherSampleTestClass extends AbstractTestClass {

    @Test
    public void testMethod1() {
    }

    @Test
    @SkipConfiguration
    public void testMethod2() {
    }
}

[krmahadevan]

@prakashnarkhede

Looking for a feature that can get rid of the extra addition of a reporting library, making test automation a little more complex, and something that focuses on providing an advanced, real-time analytics and reporting dashboard within TestNG. It aims to offer detailed insights into test execution, performance metrics, and trend analysis, facilitating better decision-making and test management.

Given that TestNG already provides you with handling TestNG results (either at the end via IReporter interface (or) on a more realtime basis via IInvokedMethodListener for e.g., I believe that you build this capability at your end without needing any additional changes into TestNG.

What you are asking for is more of a sophistication around test results analysis which I feel can be obtained if you use some sort of a tool such as https://reportportal.io/docs/

I am not sure if TestNG dev team (which is pretty lean at the moment) has the bandwidth to indulge into something like this.

@juherr - WDYT ?

[juherr]

I think reporting, except api, should be excluded from testng core.
It is an opportunity for 3rd parties and many projects already exist.

[ubutar]

What

I'd like to suggest at least some control over @DataProvider-annotated methods during retry. As discussed in #3041 dataprovider methods used to be invoked when retrying failed methods in v6, in v7 it was gradually removed until 7.8.0 when it was removed finally, so that tests are always fed the original data generated one time.

Why

• Dataproviders are frequently used as generators, which are meant to provide unique data, due to constraints of application under test. Retrying with already invalid data makes little sense
• Dataproviders with single object returned are a frequent case out there.
• It used to work like that in v6 after all
• Painful to migrate from 6 to 7

How

What I'd like to get is something similar to:

@DataProvider(alwayRun="true")
public Iterator<Object[]> method() {...}

or

@DataProvider
@Generator
public Iterator<Object[]> method() {...}

or

@Generator
public Iterator<Object[]> method() {...}

or

@DataProvider(generator="true")
public Iterator<Object[]> method() {...}

or anything which would rerun annotated method everytime a test is retried, e.g. using IRetryAnalyzer or any other retry mechanism. I guess alwaysRun=true is the most obvious choice.

For the case of NxM data object, if N1<N tests failed and are to be retried, I still suggest generating whole set of data by running the method, but pick only those data line where tests failed. For parallel execution on dataProvider level it may get tricky getting the order, but I guess internal indexing of returned objects might help (Sorry if it's already the case, I didn't check)

Things to consider

@Factory

[baflQA]

I would be happy to see more ways of controlling the Before/AfterMethod configuration.
I would e.g. want to have an easy way of skipping the first execution of the BeforeMethod, and the last execution of the AfterMethod.
E.g. - i want to refresh the page before/after the test, and it doesn't make sense to do that right after the test setup and the initial navigation, or before the cleanup and the browser shut down.

[krmahadevan]

@baflQA can you please help share a sample which contains the beforeMethod and afterMethod along with 2 or more test methods ( please include regular tests, data driven tests and tests with invocation count ) and share the expected output.

That way the example would make it very clear in terms of the expectations

[baflQA]

Hey @krmahadevan - i was thinking about something like this:
https://github.com/baflQA/testNG_parallel_debug/blob/master/src/test/java/testngparallel/SkipFirstBeforeMethodExecution.java

[krmahadevan]

@baflQA - You should be able to do this even today. Here's a sample that shows this

import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.lang.reflect.Method;
import java.util.concurrent.atomic.AtomicBoolean;

public class SampleTestCase {

    private final AtomicBoolean once = new AtomicBoolean(false);

    @BeforeMethod
    public void beforeMethod(Method method) {

        if (once.compareAndSet(false, true)) {
            //do nothing but return
            System.err.println("Skipping config method the first time for " + method.getName());
        } else {
            System.err.println("Running setup for " + method.getName());
            //simulate the setup
        }
    }

    @Test
    public void testMethod1() {}

    @Test
    public void testMethod2() {}

    @Test
    public void testMethod3() {}
}

[krmahadevan]

As a user I should be able to associate context to listener invocation.

Currently TestNG allows me to define listeners globally in a direct way (via service loaders or via the <listener> tag) (or) in an indirect way (via the annotation @Listeners). But TestNG always hoists these listeners at the global level and then applies them across the suites it finds for execution.

Maybe it would make sense for us to apply the listeners defined via the @LIsteners ONLY to the class on which it was annotated and maybe the child classes.

For e.g, lets say there's a test class that looks like below

@Listeners({DataProviderListener.class})
public class MyTestCase {
// more test code
}

TestNG should invoke the listener DataProviderListener only to MyTestCase and its sub-classes.

[X-NoNAME]

1. It would be great if we could change the status of the @ Before... methods invoke result (in the afterInvocation method)

2. Add the ability to set a timeout for config methods even if IConfigurable interface used

[bobshie]

There is an requirement we discussed before.

A DataProvider is defined that include a data set (A, B ,C ,D, E ... ) and we want to launch test cases testFunc(id) in multithreads (max thread is 3).
what we expect is:
testFunc(A) testFunc(B) testFunc(C) in parallel.
and then, after some test finished, it will create a new thread to run D, and then E, F....

interface is like:

@DataPoolProvider(max_thread = 3, dataprovider=dataset)
public void testFunc(id){
}

[krmahadevan]

The dataprovider-thread-count attribute at the suite level already lets you control the pool size. Does that not work for you? Or are you trying to control the thread pool size on a per data driven test method ?

[bobshie]

There is request to control the thread pool size based on the data. for there are some resource limit based on some data.

[krmahadevan]

@bobshie - There are some experimental features that lets you define a common thread pool for running all your data providers or running both your regular and data driven tests in a shared thread pool

See here : https://testng.org/#_controlling_threadpool_usage

PS: If you use a global thread pool in which all your regular and data driven tests will run, you may experience this issue

[bobshie]

Do we have a page that include all features that provides via TestNG specific version? I want to check what big features that TestNG 5.0 to 7.0 can provide and then I maybe use these features in my project to save some code or make test cases simple.

[krmahadevan]

@bobshie - unfortunately there's no such page that lists out features by version.

[bobshie]

Can we config invocation_count on Suite or Test level. currently, it just can configured on TestMethod level(@test).
for example:

<suite invocation_count="3">
    <test invocation_count="5">
        .....
    </test>
</suite>

[krmahadevan]

@juherr - Can you please help create the required issues that explains in detail in terms of what you have in mind around these asks.

• XSD creation
• XSD enhancements in terms of allowing undefined attributes and mechanisms to read those attributes which could be user specific.
• The migration strategy from DTD to XSD for our users
• Backward compatibility expectations which would ensure that the XSD aware TestNG honours the DTD input
• Anything else that I may have missed.

This will also help us track the list of things to be done and wrap them to closure (apart from seeking help from our users)

My understanding with respect to the xml world is very basic and so it will personally also help me understand clearly what you are envisioning

[juherr]

@krmahadevan
This feature change presents an opportunity to enhance code readability and comprehensibility.

A key goal is to separate the test model from the suite file.
Currently, the test model and suite file are intertwined, which means that anywhere the test model is utilized, an alternative object should be employed exclusively for XML processing.
To ensure backward compatibility, it may not be possible to modify the names of the model classes.

In addition, the XML processing component is somewhat outdated and could benefit from improvements such as adopting XSD instead of DTD and using JAXB instead of the lower-level SAX API.
This endeavor is dependent on the previous step and can be considered optional.

Consequently, it will become viable to modify the model and its XML representation.

[krmahadevan]

@juherr - I completely get the intent. I am more looking for a detailed guided set of steps (imagine it to be tasks in a user story) that will help me understand what all needs to be done and perhaps start work on that. I have never worked with XML processing a lot. That is why am looking for guidance here in terms of what all should we be doing. I think it will be better if we take this discussion to a dedicated issue where we can discuss.

[juherr]

@krmahadevan To be honest, I do not have a precise plan because every time I tried to implement it, time was missing to finish.
The suggested plan is as follows:

1. Convert DTD into XSD format. This conversion could be performed using appropriate tools. However, it is important to note that the conversion process should be carefully reviewed and validated. This is necessary to avoid any potential issues such as duplication and excessive permissiveness stemming from the automatic conversion.

2. Utilize JAXB to generate a structured class hierarchy based on the XSD that was created in the previous step.

3. Parse the suite files using JAXB, employing the generated class hierarchy to create instances of objects from the file content.

4. Perform a thorough evaluation to determine if there are any potential conflicts or issues when integrating both XSD and DTD.

5. Develop comprehensive documentation for end users. This documentation will serve as a helpful resource, guiding users in the process of replacing DTD with the new XSD schema.

Is it helping you?

[krmahadevan]

@juherr This adds more clarity. I will create an issue with the same content and start work on it when I get some time. Will tag you on that issue as and when I have questions

[rodionmoiseev]

Feature: Being able to run a single test method using a command-line parameter, or by using a temporary flag in the test code itself. E.g. something similar to Jest test.only (see here: https://jestjs.io/docs/api#testonlyname-fn-timeout)

Rationale:

• This is currently possible only via XML config file, which is unproductive (in the context of my usecase)

Usecase:

• I am extending an existing feature, that has a big suite of tests. Some tests are slow to run.
• I employ a test-driven development style, where I frequently make test fail, fix code until test pass, then write new failing test, fix code, and so on
• I would like to quickly run JUST ONE failing test in isolation, to avoid waiting other (successful) tests

What I need:

• Be able to run just one test
• Be able to configure TestNG as efficiently as possible to do this
  • Ideally, configuration should be in the same context as I am working, so either in the test-code, or in the test runner flag
• Needs to only be a temporary change

Hope this makes sense. Let me know what you think and/or if I am missing and existing feature that can help me here!

Note: IntelliJ UI can do this quite nicely (by internally creating a TestNG xml config). This covers some usecases, but fails to address cases when I am working on remote development environment, where I cannot leverage IntelliJ Run feature.

[krmahadevan]

My original intent is to bring this up in the spirit of contributing to the original topic of "What features would you like to see in TestNG",

And we appreciate the intent @rodionmoiseev Please dont get me wrong. I am also trying to see what all existing features can be re-used to solve different usecases. I plan to perhaps start documenting them as a how to series either within the TestNG github page itself or within the TestNG documentation portal.

So please keep the recommendations/ideas for new features coming in. It means a lot to us, that you are taking time from your schedule to help log these ideas. Appreciate it and thank you for doing that.

[bobshie]

Hello @krmahadevan , we have a requirement from tester, they want to skip some test (according to test name defined in suite file) when they run the test case, for example:
java ... TestNG --suitefile xxx.xml --skip-test "Test Demo1".
we try to use Group and TAG, it seemes that they're defined in test class and test method, we need to only skip one test by test name, not all test method in all suite file. there will be same test method in different test with different test name in suite file (with different parameters).

similar to test.skip() ??

[krmahadevan]

@bobshie when you say test name are you referring to the name attribute of the <test> tag?

[krmahadevan]

Also can you pls share some example of what you are currently seeing ( maybe as a sample project ). That way the issue is clear.

[bobshie]

I raise a new topic below.

[bobshie]

can we implement a way to skip/ignore some test from command line:

Feature: we can skip/ignore some test execute by some filter, (suite name, test name, class name or method name etc)

Example:

# three suite file
test.xml
<suite name="Suite">
    <suite-files>
        <suite-file path="./test1.xml" />
        <suite-file path="./test2.xml" />
    </suite-files>
</suite>

test1.xml
<suite name="Suite1" verbose="1">
    <test name="Demo1" preserve-order="true">
        <parameter name="name" value="name1" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo2" preserve-order="true">
        <parameter name="name" value="name2" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo3" preserve-order="true">
        <parameter name="name" value="name3" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
</suite>

test2.xml
<suite name="Suite2" verbose="1">
    <test name="Demo1" preserve-order="true">
        <parameter name="name" value="name1" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo2" preserve-order="true">
        <parameter name="name" value="name2" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
    <test name="Demo3" preserve-order="true">
        <parameter name="name" value="name3" />
        <classes>
            <class name="com.demo.Demo" />
        </classes>
    </test>
</suite>

running command:
java TestNG --suite-files test.xml -D exclude.test.list=Demo2,Demo3

the test should just execute:

suite1::Demo1(name1)
suite2::Demo1(name1)

the other class and method should be skip/ignore.
beforeclass/test/method and afterclass/test/method should be skip or ignore according to test execution list.

Usecase:

we have complex test suite include same test method with different parameters to test different traffic scenario. when some tests failed in CI automation, but we want to test mannuely, we don't want to change any soure code (test suit file and test cases, for there are in different repos), we only change the command line in CI Job to test if it works.
it's very useful in troubleshooting and debug.

[bobshie]

We tried the solution before. but there is an issue, the statistic of the test is incorrect when I use listener or the other solution.
the success,failed and ignore test number don't include the excluded test methods.

We think it's better to add the function in TestNG for their is --excludegroups options.

Thanks.

[krmahadevan]

I tried this and it works (ofcourse it would need to be enhanced to include inheritance, overridden methods handling etc.,)

import org.testng.IAlterSuiteListener;
import org.testng.internal.ClassHelper;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;

import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class ExclusionListener implements IAlterSuiteListener {

    @Override
    public void alter(List<XmlSuite> suites) {
        List<String> methods = Arrays.asList(System.getProperty("exclude", "excludeMe").split(","));
        suites.stream()
                .flatMap(it -> it.getTests().stream())
                .flatMap(it -> it.getXmlClasses().stream())
                .forEach(xmlClass -> pruneClass(xmlClass, methods));
    }

    private void pruneClass(XmlClass xmlClass, List<String> methods) {
        List<String> excluded = exclude(xmlClass, methods);
        if (!excluded.isEmpty()) {
            xmlClass.setExcludedMethods(excluded);
        }
    }

    private List<String> exclude(XmlClass xmlClass, List<String> methods) {
        List<String> toExclude = new ArrayList<>();
        for (Method m : ClassHelper.getAvailableMethodsExcludingDefaults(xmlClass.getSupportClass())) {
            if (methods.contains(m.getName())) {
                toExclude.add(m.getName());
            }
        }
        return toExclude;
    }
}

[bobshie]

we tried it before and it works. if it's provide from TestNG directly, it would be better.

[krmahadevan]

Ok. So that works. I was concerned when you said it doesn't work. So we have a way forward for this use case. That's good to know.

Now for building this I think we should list out the requirements.

Can you please create an issue and include all things that you have so far envisioned?

We should also consider and document how this new filtering mechanism would work with whatever other means that we have. That information would directly go into the documentation and so it would help to capture that details as well in the issue.

[bobshie]

Thanks, I create a new issue for it:
#3196