The at_exit code is not run when using Minitest

[akimd]

Hi!

I might be asking a FAQ, but I have not found answers to my problems. I have read several times README though, and I don't think there's more documentation elsewhere.

First, I'd like to suggest that the doc should emphasize that not only must simplecov be required early, but also that (because of Ruby's Coverage) it cannot work on the file that loads it, it will only apply to files required or loaded afterwards.

A scaled down version of my setup is as follows.

I have a lib, reduced to

# lib.rb
def do_it
  puts "done"
  42
end

Then a main file, that, in its full version, includes option handling, starting the test suite, etc.

# main.rb
require 'simplecov'
SimpleCov.start
SimpleCov.at_exit do
  puts "SimpleCov is done"
  SimpleCov.result.format!
end

require_relative 'lib'

require 'minitest'
require 'minitest/spec'

describe "check handling of dir" do
  it "should work" do
    assert_equal 42, do_it
  end
end

do_it
Minitest.run

When run, I get:

$ bundle exec ruby main.rb
67549
done
Run options: --seed 6604

# Running:

done
.

Finished in 0.000496s, 2016.1291 runs/s, 2016.1291 assertions/s.

1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

and you can see that the at_exit code was not fired, and —no surprise— there's no coverage/ created.

If I comment out the Minitest.run, then I do get

$ bundle exec ruby main.rb
67535
done
SimpleCov is done
Coverage report generated for test:units to /Users/akim/src/faveod/coverage/coverage. 384 / 986 LOC (38.95%) covered.

Now the at_exit is run, and I do get the coverage/ dir.

What do I need to do to get it to work?

Thanks a lot!

Versions

$ cat Gemfile
# frozen_string_literal: true

source 'https://rubygems.org'

ruby '>= 3.0'

gem 'minitest'
gem 'simplecov', require: false, group: :test
$ bundle install
Using bundler 2.3.7
Using docile 1.4.0
Using minitest 5.16.2
Using simplecov-html 0.12.3
Using simplecov_json_formatter 0.1.4
Using simplecov 0.21.2
Bundle complete! 2 Gemfile dependencies, 6 gems now installed.
Bundled gems are installed into `./vendor`
$ bundle exec ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [arm64-darwin21]

[stackmystack]

There's a bug with at_exit handler.

If you put SimpleCov.result.format! at the end of the script, the report will get generated.

Moreover, if you debug the same script (using rdbg) and step in SimpleCov.start, you eventually run over the at_exit handler, and theoretically it should execute the block you provided, but it doesn't.

If you invoke the handler on the stack frame, the report gets generated.

So it seems that ruby is going over it but not evaluating it.

See the screenshot:

[zenspider]

@akimd is there a special reason why you're calling Minitest.run directly?

[akimd]

Hi,

We have this script that serves as top-level for a library. It offers a bunch of options to run functions of the library, and to run selected tests. So some of its options are used to prepare the arguments passed to Minitest.run. The example above is a reduction of the real-world case.

Cheers!

[zenspider]

@akimd your code works fine if you call Minitest.autorun (or require "minitest/autorun") instead of calling Minitest.run...