Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in_tail: fix slice length to reduce memory usage #4760

Merged
merged 1 commit into from
Jan 8, 2025

Conversation

Watson1978
Copy link
Contributor

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
String#slice will allocate redundant heap area internally if an incorrect length is specified.
This PR might reduce the memory usage with in_tail plugin.

  • verify code
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'benchmark-memory'
end

str = "a" * 10_000_000
split_point = 9_999_900

Benchmark.memory do |x|
  x.report("1") {
    str.slice(split_point, str.length)
  }

  x.report("2") {
    str.slice(split_point, str.length - split_point)
  }

  x.report("3") {
    str.slice(split_point..str.length)
  }
end
  • result
Calculating -------------------------------------
                   1    10.000M memsize (    10.000M retained)
                         2.000  objects (     1.000  retained)
                         2.000  strings (     1.000  retained)
                   2    40.000  memsize (     0.000  retained)
                         1.000  objects (     0.000  retained)
                         1.000  strings (     0.000  retained)
                   3    80.000  memsize (     0.000  retained)
                         2.000  objects (     0.000  retained)
                         1.000  strings (     0.000  retained)

The all cases will generate same String object. However, 1 case will use 10 MB heap area with above code.

Docs Changes:

Release Note:

Copy link
Contributor

@kenhys kenhys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

memo:

index: 0.... k.....n-1 (total n)
k = (idx + 1) = slice_position

sliced length: 
(n-1) - k + 1
= n - k
= n - (idx + 1)
= n - slice_position

@kenhys kenhys merged commit 5080f81 into fluent:master Jan 8, 2025
12 of 13 checks passed
@Watson1978 Watson1978 deleted the fix-string-slice branch January 8, 2025 05:32
@daipom daipom added this to the v1.18.1 milestone Jan 9, 2025
@daipom
Copy link
Contributor

daipom commented Jan 9, 2025

I'm not going to backport this to LTS (v1.16.x), but if anyone has a request, please let me know.

@daipom daipom modified the milestones: v1.18.1, v1.19.0 Jan 9, 2025
kenhys added a commit that referenced this pull request Jan 29, 2025
**Which issue(s) this PR fixes**:

Backport #4750

**What this PR does / why we need it**:

It seems that it fixes failure with Ruby 3.1 on windows.

```
   undefined method `ignored?' for #<Gem::Specification:0x000001dccaf43168
```

NOTE: Initially introduced by #4681, then reverted in #4746.
Thus, it's a side-effect of #4760.

**Docs Changes**:

N/A

**Release Note**:

N/A

Signed-off-by: Kentaro Hayashi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants