Ruby Agent Testing

Introduction

Before you start, know that the Ruby test suite is massive. It covers not just the implementation of the Agent itself, but many frameworks beyond just Rails and many, many instrumentation libraries. And because so many users are still running Rails 3.x and 4.x in production and need observability, we are still testing against Rubies going back to 2.0.0 even though, at the time of this writing, Ruby 2.4 and earlier are all End of Life.

The Ruby Agent test framework is made up four primary components:

1. The unit tests most Rubyists are familiar with (i.e. rake test)
2. Mini-environments that are essentially small tests against popular frameworks (i.e. rake env:test[rails])
3. Multiverse test suites (which is all the variations of instrumentations (i.e. rake test:multiverse[httpclients])
4. Performance benchmarking tests

It is worth taking a few minutes to read this blog post to learn more about the genesis of the wide and varied testing framework we have before diving into getting tests working locally in your development environment.

Unit Tests

We run typical Test::Unit tests for the Ruby agent, which is based on minitest.  The following command runs the unit tests without Rails:

bundle exec rake test

Mini Environment Tests

If running the entire unit test suite, you should typically run it with the most recent Rails version we support. You can run against a specific Rails version only by passing the version name (which should match the name of a subdirectory in test/environments) as an argument to the test:env rake task, like this:

bundle exec rake test:env[rails51]

In CI, these unit tests are run against all supported . versions of Rails (as well as with Rails absent entirely). The test/environments directory contains dummy Rails apps for each supported Rails versions. You can also locally run tests against all versions of Rails supported by your current Ruby version with:

bundle exec rake test:env

The file parameter can be added to the test:env invocation to run a specific unit file.  It can be exact file name, or a wildcard pattern.  Multiple file patterns can be specified by separating with a comma with no spaces surrounding:

bundle exec rake test:env[rails60] file=test/new_relic/agent/distributed_tracing/*  # everything in this folder
bundle exec rake test:env[rails60] file=test/new_relic/agent/tracer_state_test.rb   # single file
bundle exec rake test:env[rails60] file=test/new_relic/agent/*_test.rb              # all *_test.rb files in this folder
bundle exec rake test:env[rails60] file=test/new_relic/agent/distributed_tracing/*,test/new_relic/agent/datastores/*  # all files in two folders

You can also run a single unit test file like this:

bundle exec ruby test/new_relic/agent_test.rb

And to run a single test within that file (note that when using the -n argument, you can either supply the entire test name as a string or a partial name as a regex):

bundle exec ruby test/new_relic/agent_test.rb -n /test_shutdown/

Functional Tests ('multiverse')

Functional tests (a.k.a. Integration tests) are run through a system called multiverse. This is very similar to running tests with appraisal. Multiverse lets us set up specific combinations of gem versions, then run a set of Test::Unit tests in that context. These can be run in the ruby agent by:

rake test:multiverse

The first time you run this command on a new Ruby installation, it will take quite a long time (up to 1 hour). This is because bundler must download and install each tested version of each 3rd-party dependency. After the first run through, almost all external gem dependencies will be cached, so things won't take as long.

Multiverse tests live in the test/multiverse directory and are organized into 'suites'. Generally speaking, a suite is a group of tests that share a common 3rd-party dependency (or set of dependencies). You can run a specific suite by providing a parameter (which gets loosely matched against suite names):

rake test:multiverse[agent_only]

Each suite consists of (at minimum) an Envfile and one or more .rb files which define tests. The Envfile specifies a set of environments in which the suite will be run. Each environment maps directly to a single Gemfile, but you'll notice that there are no Gemfiles committed to the suite directories. This is because the multiverse runner infrastructure takes care of dynamically generating Gemfiles from the Envfile when a suite is run.

Cleanup

Occasionally, it may be necessary to clean up your environment when migration scripts change or Gemfile lock files get out of sync.  Similar to Rails' rake assets:clobber, multiverse has a clobber task that will drop all multiverse databases in MySQL and remove all Gemfile.* and Gemfile.*.lock files housed under test/multiverse/suites/**

rake test:multiverse:clobber

Controlling which Tests and Environments Run

You can pass additional parameters to the test:multiverse rake task to control how tests are run:

• name= - only tests with names matching will be executed
• env= - only the Nth environment will be executed (may be specified multiple times)
• file= - only tests in will be executed
• debug - environments for each suite will be executed in serial rather than parallel (the default), and the pry gem will be automatically included, so that you can use it to help debug tests.

Example:

rake test:multiverse[agent_only,name=test_resets_event_report_period_on_reconnect,env=0,debug]

Writing Multiverse Tests

Multiverse tests are preferred when you want to write tests that have 3rd-party gem dependencies (for example, when you're testing instrumentation of a 3rd-party gem). They're also useful for writing higher-level integration-style tests in which you want to test the behavior of the agent as a whole (or multiple subsystems interacting) rather than the behavior of one specific subsystem or class.

When writing new multiverse tests, your first step should be to determine which suite they should live in. First, examine the existing suites and see if any of them might be a good fit. If your new tests have no external gem dependencies, they should probably go into the agent_only suite.  

Performance Tests

We have a suite of performance tests which exercise different parts of the codebase. Having eliminated as much noise as possible, we run a before-and-after to compare changes in memory usage and object allocations.

The following commands should be run from ruby_agent/test/performance.

To create a baseline (with dev checked out):

bundle exec ./script/runner -n test_basic_middleware_stack -B

It can be helpful to run this command a few times to make sure you have a stable baseline.

To compare against that baseline (with feature branch checked out):

bundle exec ./script/runner -n test_basic_middleware_stack -c

Again, it's worthwhile to run this a few times, and if there are variations, pick the most conservative estimates for adding to a PR discussion.

As with multiverse tests, you can run these as suites as well as individual tests. For more information on different options, see the runner file in the agent.