Gemika

Test a Ruby gem against multiple versions of everything

Gemika helps you test your gem against multiple versions of Ruby, gem dependencies and database types.

Features

Here's what Gemika can give your test's development setup (all features are opt-in):

• Test one codebase against multiple sets of runtime gem dependency sets (e.g. Rails 2.3, Rails 5.0).
• Test one codebase against multiple Ruby versions (e.g. Ruby 1.8.7, Ruby 2.3.10).
• Test one codebase against multiple database types (currently MySQL, PostgreSQL, or sqlite3).
• Compute a matrix of all possible dependency permutations (Ruby, runtime gems, database type). Manually exclude incompatible dependency permutations (e.g. Rails 5.0 does not work with Ruby 2.1).
• Let developers enter their local credentials for MySQL and PostgreSQL in a database.yml file.
• Define default Ruby version, gem dependencies and database for developers who don't care about every possible permutation for everyday work.
• Help configure a Github Actions build that tests every dependency permutation after each git push.
• Share your Ruby / gem dependeny / database permutation between local development and Github Actions.
• Define an ActiveRecord database migration that sets up your test database.
• Automatically drop and re-create your test database before each run of your test suite.
• Work around breaking changes in RSpec, Ruby and other gems

Compatibility

Gemika currently supports the following dependency versions:

• Ruby: 2.5, 2.6, 2.7, 3.2, 3.4
• RSpec: Versions 1, 2, 3
• ActiveRecord: Versions 5.2, 6.1, 7.0, 8.0
• Databases: PostgreSQL (with pg gem), MySQL or MariaDB (with mysql2 gem), or sqlite3 (with sqlite3 gem)

Gemika also makes some assumption about your Gem:

• You're testing with RSpec.
• If you use any database-related features, you need activerecord as a development dependency

Example directory structure

Below you can see the directory of a gem with a completed Gemika testing setup. The next section describes how to get there:

Gemfile.set1                                # First dependency set. Should include development dependencies and gemika.
Gemfile.set1.lock                           # Generated by `rake matrix:install`
Gemfile.set2                                # Second dependency set. Should include development dependencies and gemika.
Gemfile.set2.lock                           # Generated by `rake matrix:install`
Gemfile.set3                                # Third dependency set. Should include development dependencies and gemika.
Gemfile.set3.lock                           # Generated by `rake matrix:install`
Gemfile -> Gemfile.set2                     # Symlink to default Gemfile for development
Gemfile.lock -> Gemfile.set2.lock           # Symlink to default Gemfile.lock for development
.github/workflows/test.yml                  # Configures all tested Ruby / gemfile combinations, for both local development and Github Actions
.ruby-version                               # Default Ruby version for development
.gitignore                                  # Should ignore spec/support/database.yml
my_gem.gemspec                              # Specification for your gem
Rakefile                                    # Should require 'gemika/tasks'
README.md                                   # README for your gem
lib/my_gem.rb                               # Main file to require for your gem
lib/my_gem/my_class.rb                      # Class delivered by your gem
lib/my_gem/version.rb                       # Version definition for your gem
spec/spec_helper.rb                         # Requires 'gemika' and all files in support folder
spec/support/database.rb                    # Database schema for test database
spec/support/database.yml                   # Database credentials for local development (not checked in)
spec/support/database.sample.yml            # Sample database credentials for new developers
spec/support/database.github.yml            # Database credentials for Github Actions
spec/my_gem/my_class_spec.rb                # Tests for your gem

For a live example of this setup, check the makandra/minidusen repo.

Step-by-step integration

Have a standard gem setup

Gemika expects a standard gem directory that looks roughly like this:

my_gem.gemspec      # Specification for your gem
Rakefile            # Rake tasks for your gem
lib/my_gem.rb       # Main file to require for your gem
spec/my_gem_spec.rb # Tests for your gem

If you don't have a directory yet, you can ask Bundler to create it for you:

bundle gem my_gem

Install Gemika

Switch to your favorite Ruby version, then install Gemika:

gem install gemika

Future contributors to your gem can install Gemika using the Gemfiles we will create next.

Rake tasks

Add this to your Rakefile to gain tasks matrix:install, matrix:spec, matrix:update.

begin
  require 'gemika/tasks'
rescue LoadError
  puts 'Run `gem install gemika` for additional tasks'
end

Check that the tasks appear with rake -T:

rake current_rspec[files]  # Run specs with the current RSpec version
rake matrix:install        # Install all Ruby 1.8.7 gemfiles
rake matrix:list           # List dependencies for all Ruby 1.8.7 gemfiles
rake matrix:spec[files]    # Run specs for all Ruby 1.8.7 gemfiles
rake matrix:update[gems]   # Update all Ruby 1.8.7 gemfiles

We also recommend to make matrix:spec the default task in your Rakefile:

task :default => 'matrix:spec'

Define multiple dependency sets

For each combination of runtime and development dependencies (e.g. Rails + database), create a corresponding Gemfile in your project root.

Each Gemfile should include:

1. The runtime dependencies you'd like to test against (e.g. Rails 5)
2. The development dependencies for that set (e.g. rspec) in a version that is compatible with these runtime dependencies.
3. The gemika gem
4. Your gem under test, from path .

For instance, if one dependency set is Rails 4.2 with a MySQL database, we would create ./Gemfile.4.2.mysql2 with these contents:

source 'https://rubygems.org'

# Runtime dependencies
gem 'rails', '~>4.2.11'
gem 'mysql2', '= 0.4.10'

# Development dependencies
gem 'rspec', '~> 3.4'
gem 'rake'
gem 'byebug'
gem 'gemika'

# Gem under test
gem 'my_gem', :path => '.'

Repeat for other combinations (e.g. Rails 5.0 + PostgreSQL)

Define Ruby/Gemfile Test Matrix

Configure the test matrix in .github/workflows/test.yml, even if you're not currently running tests on Github Actions.

Use GitHub's matrix strategy:

jobs:
  my_job:
    strategy:
      matrix:
        ruby:
        - 2.1.8
        - 2.2.4
        - 2.3.1
        gemfile:
        - Gemfile.3.2.mysql2
        - Gemfile.4.2.mysql2
        - Gemfile.4.2.pg
        - Gemfile.5.0.mysql2
        - Gemfile.5.0.pg
    env:
      BUNDLE_GEMFILE: "${{ matrix.gemfile }}"
    steps:
    - uses: actions/checkout@v4
    - name: Install ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: "${{ matrix.ruby }}"
    - ...

There might be incompatible combinations of gemfiles and Rubies, e.g. Rails 5.0 does not work with Ruby 2.1 or lower. In this case, add an exclude key to your matrix in .github/workflows/test.yml

jobs:
  my_job:
    strategy:
      matrix:
        ruby:
        - 2.1.8
        - 2.3.1
        gemfile:
        - Gemfile.4.2.mysql2
        - Gemfile.4.2.pg
        - Gemfile.5.0.mysql2
        - Gemfile.5.0.pg
        exclude:
        - gemfile: Gemfile.5.0.mysql2
          ruby: 2.1.8
        - gemfile: Gemfile.5.0.pg
          ruby: 2.1.8

Alternatively, you can explicitly list all Ruby / Gemfile combinations with

matrix:
  include:
  - gemfile: Gemfile.5.0.mysql2
    ruby: 2.3.8
  - gemfile: Gemfile.5.2.mysql2
    ruby: 2.3.8

Generate lockfiles

Generate lockfiles for each Gemfile:

rake matrix:install

In this example, your project directory should now contain a lockfile for each gemfile:

Gemfile.4.2.mysql2
Gemfile.4.2.mysql2.lock
Gemfile.5.0.pg
Gemfile.5.0.pg.lock

Gemfiles and lockfiles should be committed to your repo.

Make sure to re-run rake matrix:install after each change to your gemfiles, and commit the generated changes.

Default Ruby and default gemfile

Your project will be more approachable if you're defining a default Ruby and dependency set. This way a developer can make changes and run code without knowing about the test matrix.

Create a .ruby-version file with the default Ruby version:

2.2.4

Choose a default dependency set and symlink both gemfile and lockfile to your project root:

ln -s Gemfile.4.2.mysql2 Gemfile
ln -s Gemfile.4.2.mysql2.lock Gemfile.lock

Commit both .ruby-version and symlinks to your repo.

Configure Test databases

Create a local test database (e.g. my_gem_test) for MySQL/PostgreSQL/etc.

Then add credentials to spec/support/database.yml:

mysql:
  database: my_gem_test
  host: localhost
  username: root
  password: secret

postgresql:
  database: minidusen_test
  user:
  password:

sqlite:
  database: ":memory:"

We don't want to commit our local credentials, so add a line to your .gitignore:

spec/support/database.yml

What we will commit is a database.sample.yml as a template for future contributors:

cp spec/support/database.yml spec/support/database.sample.yml

Remember to replace any private passwords in database.sample.yml with secret before committing.

To have ActiveRecord connect to the database in database.yml before your tests, add a file spec/support/database.rb with the following content:

database = Gemika::Database.new
database.connect

Now require Gemika and this support file from your spec_helper.rb.

require 'gemika'
require 'spec/support/database'

Protip: Instead of requiring support files indidually, configure your spec_helper.rb to automatically require all files in the spec/support folder:

Dir["#{File.dirname(__FILE__)}/support/*.rb"].sort.each {|f| require f}

Now you have a great place for code snippets that need to run before specs (factories, VCR configuration, etc.).

To have your database work with Github Actions, add a database file spec/support/database.github.yml.

mysql:
  database: test
  username: root
  password: password
  host: 127.0.0.1
  port: 3306

postgresql:
  database: test
  host: localhost
  username: postgres
  password: postgres
  port: 5432

Test database schema

If your gem is talking to the database, you probably need to create some example tables.

Gemika lets you define an ActiveRecord database migration for that. Before your test suite runs, Gemika will drop all tables in your test database and recreate them using this migration.

Add your migration to your spec/support/database.rb (created and required above):

database = Gemika::Database.new
database.connect
database.rewrite_schema! do

  create_table :users do |t|
    t.string :name
    t.string :email
    t.string :city
  end

  create_table :recipes do |t|
    t.string :name
    t.integer :category_id
  end

  create_table :recipe_ingredients do |t|
    t.string :name
    t.integer :recipe_id
  end

  create_table :recipe_categories do |t|
    t.string :name
  end

end

Clean database before each test

A very useful Rails default is to wrap every test in a transaction that is rolled back when the example ends. This way each example starts with a blank database.

To get the same behavior in your gem tests, add database_cleaner as a development dependency to all your gemfiles:

gem 'database_cleaner'

If you don't want to configure database_cleaner manually, you can ask Gemika to clean the database before each example:

Gemika::RSpec.configure_clean_database_before_example

Note that you also need require 'gemika' in your spec_helper.rb.

Try it out

Check if you can install development dependencies for each row in the test matrix:

bundle exec rake matrix:install

Check if you can run tests for each row in the test matrix:

bundle exec rake matrix:spec

To only run some examples, put the list of files in square brackets (it's a Rake thing):

bundle exec rake matrix:spec[spec/foo_spec.rb:1005]

You should see the command output for each row in the test matrix. Gemika will also print a summary at the end:

If you now discover compatibility issue with your library, see below how Gemika can help you bridge incompatibilities between dependency sets.

Running specs in multiple Ruby versions

Note that there is no task for automatically running all gemfiles in all Ruby versions. We had something like this in earlier versions of Gemika and it wasn't as practical as we thought.

Instead you need to manually switch Ruby versions and re-run:

rake matrix:install
rake matrix:spec

Note that if your current Ruby version is very far away from your default Ruby in .ruby-version, you might need to run rake with a gemfile that has compatible dependencies:

BUNDLE_GEMFILE=Gemfile.2.3 bundle exec rake matrix:install
BUNDLE_GEMFILE=Gemfile.2.3 bundle exec rake matrix:spec

We recommend to setup Github Actions to check the entire test matrix after each push, including all Rubies. This way developers can stay on the default Ruby and gemfile most of the time while the pipeline checks make sure that nothing broken gets merged.

Activate Github Actions

We recommend to setup Github Actions to check the entire test matrix after each push. This will also show the test results on a pull request's page, helping maintainers decide whether a PR is safe to merge.

Add development instructions to your README

Your README should contain instructions how to run tests before making a PR. We recommend to add a section like the one below to your README.md:

## Development

There are tests in `spec`. We only accept PRs with tests. To run tests:

- Install Ruby x.y.z
- Create a local test database `my_gem_test` in both MySQL and PostgreSQL
- Copy `spec/support/database.sample.yml` to `spec/support/database.yml` and enter your local credentials for the test databases
- Install development dependencies using `bundle install`
- Run tests using `bundle exec rspec`

We recommend to test large changes against multiple versions of Ruby and multiple dependency sets. Supported combinations are configured in `.github/workflows/test.yml`. We provide some rake tasks to help with this:

- Install development dependencies using `bundle matrix:install`
- Run tests using `bundle matrix:spec`

Note that we have configured Github Actions to automatically run tests in all supported Ruby versions and dependency sets after each push. We will only merge pull requests after a green build.

Adjust the first part to match what you chose as your default Ruby and default gemfile.

Bridging incompatibilities between dependency sets

Gemika can help you bridge incompatibilities or breaking changes between Ruby versions, gem versions, or RSpec.

Version switches

Check if a gem was activated by the current gemfile:

Gemika::Env.gem?('activesupport')

Check if a gem was activated and satisfies a version requirement:

Gemika::Env.gem?('activesupport', '>= 5')
Gemika::Env.gem?('activesupport', '~> 5.0.0')
Gemika::Env.gem?('activesupport', '< 5')

Check if the current Ruby version satisfies a version requirement:

Gemika::Env.ruby?('>= 2')
Gemika::Env.ruby?('< 2')
Gemika::Env.ruby?('~> 2.1.0')

Check if the process is running as a Github Actions test

Gemika::Env.github?

RSpec 1 vs. RSpec 2+

If you're testing gems against Rails 2.3 or Ruby 1.8.7 you might need to test with RSpec 1. There are a lot of differences between RSpec 1 and later versions, which Gemika helps to pave over.

Configuring RSpec requires you to work on a different module in RSpec 1 (Spec::Runner) and RSpec 2 (just RSpec). The following works for all RSpec versions:

Gemika::RSpec.configure do |config|

  config.before(:each) do
    # runs before each example
  end

end

When your tests need to run with RSpec 1, you need to use the old should syntax, which works across all RSpec versions.

To enable this should syntax for later RSpecs:

Gemika::RSpec.configure_should_syntax

RSpec 1 has a binary spec, while later RSpecs use rspec. To call the correct binary for the current gemfile:

rake current_rspec

Development

Here are some hints when you try to make changes to Gemika itself:

There are tests in spec. We only accept PRs with tests. If you create a PR, the tests will automatically run on GitHub actions on each push. We will only merge pull requests after a green GitHub actions run.

To run tests locally for development, first setup your test databases:

• Create a local test database gemika_test in both MySQL and PostgreSQL
• Copy spec/support/database.sample.yml to spec/support/database.yml and enter your local credentials for the test databases

Afterwards you have multiple options:

1. Run tests against the "main development" Ruby version (.ruby-version) and dependencies (Gemfile/Gemfile.lock symlinks):

   • Install the Ruby version specified in .ruby-version
   • Install development dependencies using bundle install
   • Run tests using bundle exec rspec

2. Run tests against a specific Ruby version (out of those mentioned in .github/workflows/test.yml) and all Gemfiles compatible with that version:

   • Install and switch to the Ruby version
   • Install development dependencies for all compatible Gemfiles using rake matrix:install
   • Run tests for all compatible Gemfiles using rake matrix:spec

3. Run tests against all compatible combinations of Ruby and Gemfile:

   • Install all Ruby versions mentioned in .github/workflows/test.yml
   • run bin/matrix (only supports rbenv for switching Ruby versions currently)

Hints:

• We recommend to have sufficiently new versions of bundler (> 2.3.0) and rubygems (> 3.3.0) installed for each Ruby version.
• The script bin/matrix will warn you, if that is not the case. For all other methods you need to ensure that yourself.
• Supported "Ruby <-> Gemfile" combinations are configured in .github/workflows/test.yml.

Credits

Henning Koch from makandra