Memory Bloat

[M-Dahab]

Hi,

I have a rails application that defines dynamic Mongoid models and I'm facing a memory bloat issue where Ruby GC skip model instances, and within 100 requests, memory usage - of ruby process - jumps to ~600M of RAM and never drops again.

Here's a script to "reproduce" the issue:

# Ruby 2.6.7p197 (2021-04-05 revision 67941) [x86_64-darwin20]
# Mongoid 7.3.0

require 'mongoid'

Mongoid.configure do |config|
  config.clients.default = {
    hosts: ['localhost:27017'],
    database: 'memory_bloat',
  }

  config.log_level = :warn
end

class Base
  def self.inherited(sub_class)
    super

    sub_class.class_eval do
      include Mongoid::Document
      include Mongoid::Timestamps
    end
  end
end

class Company < Base
  embeds_many :departments

  attr_reader :employee_models

  after_initialize do
    @employee_models = departments.map do |department|
      Object.send :remove_const, department.employee_model_name if Object.const_defined? department.employee_model_name

      Object.const_set(
        department.employee_model_name,
        Class.new(Base) do
          store_in collection: "d#{department.id}_employees"

          department.employee_fields.each do |employee_field|
            field employee_field.name, type: employee_field.value_type
          end
        end,
      )
    end
  end

  field :name, type: String
end

class Department < Base
  embedded_in :company
  embeds_many :employee_fields

  field :name, type: String

  def employee_model_name
    "D#{id}Employee"
  end
end

class EmployeeField < Base
  embedded_in :department

  field :name, type: :stringified_symbol
  field :value_type, type: :stringified_symbol
end

Company.create!(
  name: 'Company1',
  departments: [
    {
      name: 'IT',
      employee_fields: [
        { name: 'name', value_type: 'string' },
        { name: 'email', value_type: 'string' },
        { name: 'server_room_access', value_type: 'boolean' },
      ],
    },
    {
      name: 'HR',
      employee_fields: [
        { name: 'name', value_type: 'string' },
        { name: 'email', value_type: 'string' },
        { name: 'payroll_access', value_type: 'boolean' },
      ],
    },
  ],
)

puts 'Model instance count:'
puts(ObjectSpace.each_object(Base).each { |i| p [i.class, i.id.to_s] })

Company.last
Company.last
Company.last

puts "\nModel instance count after 3 finds:"
puts(ObjectSpace.each_object(Base).each { |i| p [i.class, i.id.to_s] })

GC.start

puts "\nModel instance count after GC:"
puts(ObjectSpace.each_object(Base).each { |i| p [i.class, i.id.to_s] })

Mongoid.purge!

The output for me is:

Model instance count:
[EmployeeField, "6191fe4253d98d37710aba95"]
[EmployeeField, "6191fe4253d98d37710aba94"]
[Department, "6191fe4253d98d37710aba9a"]
[EmployeeField, "6191fe4253d98d37710aba97"]
[Department, "6191fe4253d98d37710aba9b"]
[EmployeeField, "6191fe4253d98d37710aba96"]
[Company, "6191fe4253d98d37710aba9c"]
[EmployeeField, "6191fe4253d98d37710aba99"]
[EmployeeField, "6191fe4253d98d37710aba98"]
[D6191fe4253d98d37710aba9bEmployee, "6191fe4253d98d37710aba9e"]
10

Model instance count after 3 finds:
[EmployeeField, "6191fe4253d98d37710aba99"]
[EmployeeField, "6191fe4253d98d37710aba98"]
[EmployeeField, "6191fe4253d98d37710aba97"]
[EmployeeField, "6191fe4253d98d37710aba99"]
[EmployeeField, "6191fe4253d98d37710aba98"]
[EmployeeField, "6191fe4253d98d37710aba97"]
[Department, "6191fe4253d98d37710aba9b"]
[Department, "6191fe4253d98d37710aba9a"]
[Company, "6191fe4253d98d37710aba9c"]
[Department, "6191fe4253d98d37710aba9a"]
[Department, "6191fe4253d98d37710aba9b"]
[Department, "6191fe4253d98d37710aba9a"]
[Company, "6191fe4253d98d37710aba9c"]
[Company, "6191fe4253d98d37710aba9c"]
[Department, "6191fe4253d98d37710aba9b"]
[EmployeeField, "6191fe4253d98d37710aba96"]
[EmployeeField, "6191fe4253d98d37710aba95"]
[EmployeeField, "6191fe4253d98d37710aba94"]
[EmployeeField, "6191fe4253d98d37710aba96"]
[EmployeeField, "6191fe4253d98d37710aba95"]
[EmployeeField, "6191fe4253d98d37710aba94"]
[EmployeeField, "6191fe4253d98d37710aba96"]
[EmployeeField, "6191fe4253d98d37710aba95"]
[EmployeeField, "6191fe4253d98d37710aba94"]
[EmployeeField, "6191fe4253d98d37710aba99"]
[EmployeeField, "6191fe4253d98d37710aba98"]
[EmployeeField, "6191fe4253d98d37710aba97"]
27

Model instance count after GC:
[Department, "6191fe4253d98d37710aba9a"]
[Company, "6191fe4253d98d37710aba9c"]
[Department, "6191fe4253d98d37710aba9b"]
[EmployeeField, "6191fe4253d98d37710aba96"]
[EmployeeField, "6191fe4253d98d37710aba95"]
[EmployeeField, "6191fe4253d98d37710aba94"]
[EmployeeField, "6191fe4253d98d37710aba99"]
[EmployeeField, "6191fe4253d98d37710aba98"]
[EmployeeField, "6191fe4253d98d37710aba97"]
9

However, if you comment out after_initialize in Company the issue disappears:

Model instance count:
[Company, "6191ffe453d98d3e7b5fa717"]
[Department, "6191ffe453d98d3e7b5fa715"]
[EmployeeField, "6191ffe453d98d3e7b5fa70f"]
[EmployeeField, "6191ffe453d98d3e7b5fa711"]
[EmployeeField, "6191ffe453d98d3e7b5fa710"]
[EmployeeField, "6191ffe453d98d3e7b5fa712"]
[Department, "6191ffe453d98d3e7b5fa716"]
[EmployeeField, "6191ffe453d98d3e7b5fa714"]
[EmployeeField, "6191ffe453d98d3e7b5fa713"]
9

Model instance count after 3 finds:
[Company, "6191ffe453d98d3e7b5fa717"]
[Company, "6191ffe453d98d3e7b5fa717"]
[Company, "6191ffe453d98d3e7b5fa717"]
[Company, "6191ffe453d98d3e7b5fa717"]
[Department, "6191ffe453d98d3e7b5fa715"]
[EmployeeField, "6191ffe453d98d3e7b5fa70f"]
[EmployeeField, "6191ffe453d98d3e7b5fa711"]
[EmployeeField, "6191ffe453d98d3e7b5fa710"]
[EmployeeField, "6191ffe453d98d3e7b5fa712"]
[Department, "6191ffe453d98d3e7b5fa716"]
[EmployeeField, "6191ffe453d98d3e7b5fa714"]
[EmployeeField, "6191ffe453d98d3e7b5fa713"]
12

Model instance count after GC:
0

I have also tried to memic my app and create some records for the dynamic employee models:

Company.last.employee_models.first.create!(
  name: 'IT Employee1',
  email: '[email protected]',
  server_room_access: true,
)
Company.last.employee_models.last.create!(
  name: 'HR Employee1',
  email: '[email protected]',
  payroll_access: false,
)

But that doesn't increase the count and those records are garbage collected.

One last thing to mention is that in my app it's much worse, and even after GC the count stays the same.
Also, this script produces different initial count (first count after Company.create! jumps between 10, 19 and 25 randomly and maybe more/less 🤷🏾‍♂️) for each run!

Thank you.

[p-mongo]

Hi @M-Dahab ,

I ran your reproduce case and it does reproduce the issue but only on Ruby 2.6. On Ruby 2.7 and 3.1 all models get garbage collected. We recommend you upgrade your Ruby version to solve this issue.

[M-Dahab]

Yes, thank you!
Upgrading to Ruby 2.7.0 fixed the issue.