JUnit4 Reporter for Instrumentation Tests – Possible?

[brunnoferreira]

Hi there!

I have been trying to set up a project to report test results directly to Qase, but can't seem to make it work. We are running integration tests using a custom AndroidJUnitRunner (so JUnit4) with a mix of Espresso and Compose tests.

The guide I have been using is the one for the JUnit4 Reporter here: https://github.com/qase-tms/qase-java/tree/main/qase-junit4-reporter. More specifically, the Gradle parts of it.

These being instrumentation tests, though, there are a couple of details on that guide that don't quite fit.

• Task type is not Test, but rather DeviceProviderInstrumentTest, where systemProperties is not available. That means this line does not apply:
  test { systemProperties = System.properties }

• Running on a device/emulator, there's no JVM to be configured. This also makes it impossible to complete the aspectjweaver setup:
  test.doFirst { useJUnit() def weaver = configurations.aspectjweaver.find { it.name.contains("aspectjweaver") } jvmArgs += "-javaagent:$weaver" }

Is there anything that can be done about those? Is the reporter even supported for instrumentation tests or is there a different guide that I should be following instead?

I have also been trying to debug my implementation so far, but haven't been able to. I saw suggestions to set the debugging mode on the QaseClient (like in #106 (comment)), but the guide I'm following doesn't make use of that dependency so I don't have it available at all in my project.

Thanks in advance for any help!

[gibiw]

Hi there!

I'm not sure if our reporter will work with a custom runner.

1. The line test { systemProperties = System.properties } is used to pass command-line arguments specified at runtime into the test environment.
   If you're not using such arguments, you can skip adding this.

2. Aspects are required for the @Step annotation and for automatically integrating the reporter with the standard runner. I believe this might not be relevant for your case either.

3. Regarding the question about QaseClient, it refers to an old version that is no longer supported. It has now been split into two separate parts:

   • qase-api-client: An API client for interacting with Qase TMS.
   • qase-java-commons: A shared package for all Java reporters, which contains the core logic for working with Qase TMS.

Here are some options you might want to explore:

1. We have a reporter for Kaspresso, which works with Kotlin and Android tests. You can check it out here: qase-kaspresso-reporter
2. You could also try using a third-party reporter that generates results in the JUnit XML format and upload them using our CLI utility: qasectl

[brunnoferreira]

Hey @gibiw, thanks for the quick reply!

I will continue investigating those tomorrow based on your observations, thanks so much for that. I just had a couple of notes/quick follow up questions to guide my next steps:

• Maybe custom runner wasn't the correct wording – what we have is simply a subclass of AndroidJUnitRunner (default runner, I believe) that does very little, and is set using the testInstrumentationRunner property inside the android Gradle configuration block. Are you saying that just by specifying a runner like that we wouldn't be able to use the JUnit4 reporter?
• Am I correct in assuming that if we went down another path like using the CLI utility or even trying the REST API directly (https://developers.qase.io/reference/introduction-to-the-qase-api – do I understand it correctly, can it be used for reporting test results?) a third party reporter would be required? JUnit output can never be uploaded directly?

[gibiw]

@brunnoferreira I cannot give you a definitive answer to your question about the custom runner. To understand how it works, I would need to review the code. Could you provide more details? For example, a test project.

[brunnoferreira]

I started setting up a sample project but realised I couldn't even get Unit Tests to be uploaded on Android, so maybe let's start with that so I can figure out what is wrong?

I have attached a test project here, hopefully it is straightforward. I added a qase.config.json and modified my build.gradle.kts file accordingly. I have based the setup on the instructions linked previously and the example Gradle JUnit4 project here: https://github.com/qase-tms/qase-java/tree/main/examples/junit4/junit4-gradle.

Settings should be the equivalent of what we have in the example project – except using the Kotlin DSL – minus the Java part. When updating the config json file in the example with an API Token and a project code I see test runs uploaded to Qase, but when doing the same in the test project attached here, nothing happens.

QaseSample.zip