OpenSSL::Cipher::CipherError under high concurrent load.

[omaik]

Hey there!

Having the following exception on a random basis

OpenSSL::Cipher::CipherError: null
  from encryptor.rb:98:in `final'
  from encryptor.rb:98:in `crypt'
  from encryptor.rb:49:in `decrypt'
  from attr_encrypted.rb:240:in `decrypt'
  from attr_encrypted.rb:329:in `decrypt'
  from attr_encrypted.rb:161:in `block (2 levels) in attr_encrypted'

Steps to Reproduce

1. gem version

    attr_encrypted (3.1.0)
      encryptor (~> 3.0.0)

2. model

class Customer < ApplicationRecord
  ...
  attr_encrypted :ssn, key: Settings.data_encryption.encryption_key
  ...
end

3. test sidekiq worker

class ConcurrencyWorker
  include Sidekiq::Worker

  def perform
    customer = Customer.last
    customer.assign_attributes(ssn: SecureRandom.hex)
  end
end

4. start sidekiq with 60 threads

bundle exec sidekiq -c 60

5. from Rails Console

> loop { ConcurrencyWorker.perform_async }

You will see the exception above once per 3-5 minutes.

Additional details(may give a clue)

Played with retrying this error

The following approach Does not work, every consequent retry fails with the same error.

class ConcurrencyWorker
  include Sidekiq::Worker

  def perform
    customer = Customer.last
    begin
      retry_count ||= 0
      customer.assign_attributes(ssn: SecureRandom.hex)
    rescue OpenSSL::Cipher::CipherError => e
      retry_count += 1
      Sidekiq.logger.info('Retrying')
      raise e if retry_count > 3

      retry
    end
  end
end

The following approach Does work. The job succeeds after the first retry.

class ConcurrencyWorker
  include Sidekiq::Worker

  def perform
    retry_count ||= 0
    customer = Customer.last
    customer.assign_attributes(ssn: SecureRandom.hex)
  rescue OpenSSL::Cipher::CipherError => e
    retry_count += 1
    Sidekiq.logger.info('Retrying')
    raise e if retry_count > 3

    retry
  end
end

That gives the assumption that state corruption happens on an individual ActiveRecord object level.

The issue seems to be reproducible only in a multithreaded environment under high load.

Thanks in advance for any help provided :)

[bblack]

d4ca0e2 fixed this for me.

[jpcamara]

d4ca0e2 fixed this for me.

Fixed it for me as well