Stale connection in DatabaseCleaner::ActiveRecord::Truncation#connection

[dmolesUC]

Summary

Each instance of the DatabaseCleaner::ActiveRecord::Truncation strategy initializes its @connection field only once, using the ActiveRecord default connection. When the connection is first requested, ActiveRecord checks it out of the connection pool. Various hooks (e.g. ActiveRecord::TestFixtures.teardown_fixtures) can return the connection to the pool, but Truncation is unaware of this and holds onto the connection object.

If another thread checks the same connection out of the pool and calls disconnect!, then when Truncation tries to use the connection to clean the database, it will find that the connection is closed and raise an error.

Steps to reproduce:

1. In a Rails/PostgreSQL project using RSpec, configure DatabaseCleaner as follows:

   RSpec.configure do |config|
     config.before(:suite) do
       DatabaseCleaner.strategy = :truncation
     end
   
     config.around do |example|
       DatabaseCleaner.cleaning do
         example.run
       end
     end
   end

2. Write a test that, in a background thread, checks a connection out of the pool, removes it from the pool, and disconnects it, e.g.

    describe 'connection pooling' do
      def do_disconnect
        Thread.new do
          connection_pool = ActiveRecord::Base.connection_pool
          connection = connection_pool.checkout.tap do |conn|
            connection_pool.remove(conn)
          end
          begin
            connection.execute('SELECT 1')
          ensure
            connection.disconnect!
          end
        end
      end
    
      5.times do |i|
        it "test #{i}" do
          ActiveRecord::Base.connection.execute('SELECT 1')
          sleep(0.5)
          do_disconnect if i % 2 == 0
        end
      end
    end

   (This example is obviously quite contrived; I ran into the problem in a more realistic situation. See discussion in Fix flakey async test thread error bensheldon/good_job#849.)

Expected

• Tests pass.

Actual

• First couple of tests pass
• Subsequent tests fail with ActiveRecord::ConnectionNotEstablished: connection is closed raised from DatabaseCleaner.cleaning via Truncation#clean

Workaround

Instead of using DatabaseCleaner.cleaning in an around block, explicitly call DatabaseCleaner.clean_with(:truncation) in an after(:each) block:

RSpec.configure do |config|
  config.after(:each) do
    DatabaseCleaner.clean_with(:truncation)
  end
end

Proposed fix

Get a fresh connection in each call to Truncation.clean -- I simulated this with a prepended module and it seems to work:

module Cleaninator
  def clean
    @connection = nil
    super
  end
end

class DatabaseCleaner::ActiveRecord::Truncation
  prepend Cleaninator
end

[olehvavryniv]

@dmolesUC Thanks a lot! Your workaround saved my life! We use DatabaseCleaner in test API for end-to-end testing. Same issue - connection is closed after ~5 mins.