-
Notifications
You must be signed in to change notification settings - Fork 11
Local or Instrumented tests
The Android platform is traditionally very hard to test. Since the environment in which we code is a different OS than the one our application will run, traditional testing practices does not apply very well to Android development. So, the approaches we have are the following:
- Deploy the application we want to test and send "commands" to it as if it were an user
- Make a clone of the environment that runs under the same environment we code
The way we can send to commands to an application during tests is through another application that "instruments" the one we intend to assert on. These are always deployed tests, that is: they are run in a real device or emulator.
The platform has always provided the first approach with the name "instrumentation tests". Those "commands" are run by an instrumentation framework. You have classes like android.test.ActivityInstrumentationTestCase2 or android.test.InstrumentationTestCase. These provide the base API for creating tests with instrumentation commands.
When you run these tests it is important to know that in reality you are deploying two applications: your real application and the test application. The later is the code you put under androidTest folder. These are whole APKs and not only the test classes. That means they have resources, AndroidManifest.xml, assets and classes.dex in them.
Nowadays, Google does not advise us to use the android.test package directly but rather the support test libraries. They contain several enhancements over the normal package including a more recent compatibility with the JUnit 4 testing Java library. This also brings android.support.test.InstrumentationRegistry which is a facility class to access the instrumentation context. The methods there, though, may seem a bit odd at first.
In android.support.test.InstrumentationRegistry we have:
-
getContext(): contrary to what it might seem, this is not the main application context. This is the test application context. Through it we can access the assets, resources and etc of the test application. Remember that in this kind of test we actually deploy two complete APKs and this is the context that references our test APK. -
getTargetContext(): this is the application context. The application we are testing.
Beware of the difference. If you are running into issues like it can't find the asset then be certain you are using the proper context.
Another kind of tests in Android is when we run pure Java tests. In Android, we don't actually execute any Java bytecode. Java is a language that compiles to an intermediate instruction that is understood by the runtime, in this case, the Java Virtual Machine. Android is similar but different. It has its own virtual machine called Dalvik or ART (Android Runtime) that understands a different bytecode.
The fact that both virtual machines understand different bytecodes makes things a bit more complicated. Sometimes we want to run tests on the JVM (Java Virtual Machine) because it is faster than deploying two whole APKs and let the system start a context for each. Running the tests on the same JVM that our development environment usually makes things orders of magnitude faster than instrumented tests.
In order to do that we can try to make our classes free of any Android dependencies using things like MVP or other architectures. This usually boils down to a great deal of bureaucracy in code. Having several interfaces and classes that its sole purpose are to delegate implementations. When we have classes that are not dependent on Android then we can test it on a JVM without problems.
But what happens if we do have a dependency and it gets called? You will receive a RuntimeException with a message "Stub!". So, dead end?
Not really! The alternative is to make the bytecode compatible! How? Well, let's re-implement the whole Android stack for our tests :) Well, we can skip some parts and just propagate others to real Java counter parts. The fact is: there is already a library that pursuits just that. Robolectric is your friend.
So, when using Robolectric we can run tests locally on the JVM as if they were normal Java code being tested. In the end, inside your application, the virtual machine will be a different one and this is crucial. Don't forget that! Things might go wrong in your JVM tests that simply won't fail on real Android. Things like cryptography are really different amongst platforms.
Well, that is the golden question right? As is the case with those: there is no right answer. Normally, instrumentation tests are focused on user interaction assertions. That is, you send commands as if you were performing what a real user of the application would try. Then you make assertions based on the results.
On the other hand, JVM tests are focused on method invocation assertions. That is, you execute a given method with given inputs and assert on outputs. You don't need the whole lifecycle of things or handling of UX (user experience) events. You just shoot with several inputs and expect some outputs.
As stated, the first approach is better suited for interface testing. What happens to the UX if some button is clicked or if some list is scrolled? That is normally what you want to test with instrumentation.
General Java tests want to find bad logic in your method implementations. This is often logic bound. You want to check if your interests calculation is taking care of all corner cases.
Normally, Android applications should not carry much logic in them. They are the front end of some service. Examples are: Gmail, Facebook, Spotify and etc. They have a lot of code to deal with UX and not that much to deal with business logic. That is because business is handled in the backend, not the frontend. So, generally, there should be more instrumentation tests than JVM tests.
That is not to say that instrumentation tests are always preferable. They do carry some issues with them. For instances: they are hard to make non flaky. Instability is a general concern with them as they can have memory issues, external apps dependencies and so on. In the past, this made them almost impossible to have. Current tools like Espresso and the InstrumentationTestRunner make them a lot more reliable. Though not as much as JVM tests.
It provides two rules: InstrumentedTestRequestMatcherRule for instrumented tests and LocalTestRequestMatcherRule for local JVM tests.
Absolutely not! They are also crucial in the quality gateway of a complete product. First, let's make it clear what are we refering to:
One thing is unit testing where you nail down all possibilities of a single unit of your code or your UI. This is normally the target of our instrumentation and local tests in Android. We don't normally test a whole scenario just to be sure that clicking a button gives us the expected message. We can simply start the screen we want (Activity, Fragment or View) and perform the action directly without a scenario context.
That is not the way BDD or SBE usually works. They have a different target that is the user story. You normally have a complete end-to-end scenario that works just the same as the user would do in the application. What is really nice about them is that focus on a user story allows them to be cross channel. That means: probably the same story should be used to test iOS, Android and Web applications. When something goes badly the message should be consistent amongst platforms. We can't do that with instrumentation or local tests in Android.
Ok, now a silver question (golden has been taken already): should I write tests if we have BDD or SBE?
You probably already know the answer. Of course we must write tests! As stated on the last point, tests we write are different than BDD or SBE. We are in full control of a unit of code or interaction. That means you can grab objects and ensure they are in the state you want them to be. That means you don't need an API for ensuring your code will handle failure of external dependencies in a nice way.
Aaaaaand that is where RequestMatcher comes in! To ease testing code that calls external APIs. The facilities are:
- Ensure your code calls the APIs the number of times you expect them to.
- Ensure your requests are properly generated.
- Help you write tests that are understood by others ("this interaction or method will need these fixtures").