Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐞: TestNg throws an error when trying to run dependent tests with withAllureUpload(){} #1017

Open
1 task done
IspecialQ opened this issue Mar 12, 2024 · 2 comments
Open
1 task done
Labels
triage type:bug Something isn't working

Comments

@IspecialQ
Copy link

What happened?

TestNg throws an error when trying to run dependent tests with withAllureUpload(){}

I know that dependent tests violate idempotency and atomicity of tests, but we have a product with complex logic and sometimes it is necessary to make tests dependent, so we use the TestNg framework, which has parameters like dependsOnMethods or dependsOnGroups.

We have integrated with Allure TestOps, but we also want to use the ability of selective test execution. After studying the TestOps documentation, we tried to dynamically generate testng.xml based on testplan.json, which is created by TestOps during selective test execution, so that all tests in the class where the necessary test is located are launched.

The generator gets the full method names from testplan.json, cuts off the method name, and writes the full class name. According to our idea, the results should be updated for both dependent and independent tests in class.

An example of the generated testng.xml file:


<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite thread-count="1" name="Generated TestNg Suite">

<test thread-count="1" name="Generated Test" group-by-instances="true" verbose="5">

<classes>

<class name="ru.sbt.snuil.somemicroservice.test.SomeMicroserviceTest">

<methods>

<include name=".*"/>

</methods>

</class>

</classes>

</test> <!-- Generated Test -->

</suite> <!-- Generated TestNg Suite -->

But when we run such a testng.xml file, we still get an error:

Method "SomeClassTest.someTest1()[pri:0]" depends on nonexistent method "someTest2"

Some mechanism, possibly filtering, breaks the launch of tests in the order we need.We would like to generate testng.xml ourselves based on testplan.json and have it run smoothly and publish results in TestOps. Maybe it can be disabled or do you have a solution for selective execution of dependent tests for TestNg?

What Allure Integration are you using?

allure-testng

What version of Allure Integration you are using?

2.20.0

What version of Allure Report you are using?

2.20.0

Code of Conduct

  • I agree to follow this project's Code of Conduct
@IspecialQ IspecialQ added triage type:bug Something isn't working labels Mar 12, 2024
@baev
Copy link
Member

baev commented Mar 12, 2024

To disable the Allure tests filter, you must unset the following environment variables: ALLURE_TESTPLAN_PATH and AS_TESTPLAN_PATH before you run the tests.

Also, we can support dependent tests in TestNG:

The filtering is done in https://github.com/allure-framework/allure-java/blob/main/allure-testng/src/main/java/io/qameta/allure/testng/AllureTestNgTestFilter.java class.

We need to analyse all the test methods and include all the methods that selected tests depend on in the run. That should do the trick.

Although, what should we do with groups? If some selected test depends on the group, should we include all the tests in such a group?

@IspecialQ
Copy link
Author

I tried using 'unset ALLURE_TESTPLAN_PATH and AS_TESTPLAN_PATH before tests run'.

As far as I know, an environment variable cannot be completely unset.

I tried like:


env.ALLURE_TESTPLAN_PATH=''

env.AS_TESTPLAN_PATH=''

 

env.ALLURE_TESTPLAN_PATH=null

env.AS_TESTPLAN_PATH=null

 

sh 'export ALLURE_TESTPLAN_PATH=""; export AS_TESTPLAN_PATH=""'

and different variations with unset, but the result the same. Here is an example of the console log:


[Allure] [320683] Job run [https://testops.example.ru/jobrun/601151] started....

[Allure] [320683] Watching on [allure-results] for session [731274]

[Allure] [320683] Save test plan for [1] tests

[Allure] [320683] Test plan with [1] tests created at [/u01/jenkins/somePath/.allure-ee/testplan.json]

+ wget -q http://someserver/testng-xml-genarator.jar   **-----//here we download our testng generator**

+ java -jar testng-xml-genarator.jar '/somePath/.allure-ee/testplan.json'  **-----//here we generate testng.xml**

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite thread-count="1" name="Generated TestNg Suite">

  <test thread-count="1" name="Generated Test" group-by-instances="true" verbose="5">

    <classes>

      <class name="ru.sbt.somemicroservice.test.SomeTest">

        <methods>

          <include name=".*"/>

        </methods>

      </class>

    </classes>

  </test> <!-- Generated Test -->

</suite> <!-- Generated TestNg Suite -->

+ unset ALLURE_TESTPLAN_PATH   **-----//here was different ways to unset variables**

+ unset AS_TESTPLAN_PATH   **-----//here was different ways to unset variables**

+ java -ea -javaagent:lib/aspectjweaver-1.9.7.jar -Dfile.encoding=UTF-8 -jar lib/somemicroservice-tests.jar -d test-output testng/testng_custom_suite.xml

===== Invoked methods

=====

[09:42:52] [n] r.l.LoggingTestListener   ================================

[09:42:52] [n] r.l.LoggingTestListener   ================================

[09:42:52] [n] r.l.LoggingTestListener   TEST RESULTS (timings, errors):

 

===============================================

    Generated Test

    Tests run: 0, Failures: 0, Skips: 0

===============================================

 

org.testng.TestNGException:

Method "SomeTest.SomeTest2()[pri:0, instance:ru.sbt.someMicroservice.test.SomeTest@65be88ae]" depends on nonexistent method "ru.sbt.someMicroservice.test.SomeTest.someTest1"

It would be great if you could implement "We need to analyze all the test methods and include all the methods that the selected tests depend on in the run."

About groups, we use them very rarely, but I agree that if a test is related to a group, then the entire group should be run - otherwise, the test result will be incorrect/failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triage type:bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants