[ruby] Upgrade Ruby to 3.2.0 #740

depfu · 2022-12-27T07:16:56Z

Here is everything you need to know about this upgrade. Please take a good look at what changed and the test results before merging this pull request.

What changed?

Release Notes

3.2.0

Posted by naruse on 25 Dec 2022

We are pleased to announce the release of Ruby 3.2.0. Ruby 3.2 adds many features and performance improvements.

WASI based WebAssembly support

This is an initial port of WASI based WebAssembly support. This enables a CRuby binary to be available on a Web browser, a Serverless Edge environment, or other kinds of WebAssembly/WASI embedders. Currently this port passes basic and bootstrap test suites not using the Thread API.

Background

WebAssembly (Wasm) was originally introduced to run programs safely and fast in web browsers. But its objective - running programs efficiently with security on various environment - is long wanted not only for web but also by general applications.

WASI (The WebAssembly System Interface) is designed for such use cases. Though such applications need to communicate with operating systems, WebAssembly runs on a virtual machine which didn’t have a system interface. WASI standardizes it.

WebAssembly/WASI support in Ruby intends to leverage those projects. It enables Ruby developers to write applications which run on such promised platforms.

Use case

This support encourages developers to utilize CRuby in a WebAssembly environment. An example use case is TryRuby playground’s CRuby support. Now you can try original CRuby in your web browser.

Technical points

Today’s WASI and WebAssembly itself is missing some features to implement Fiber, exception, and GC because it’s still evolving, and also for security reasons. So CRuby fills the gap by using Asyncify, which is a binary transformation technique to control execution in userland.

In addition, we built a VFS on top of WASI so that we can easily pack Ruby apps into a single .wasm file. This makes distribution of Ruby apps a bit easier.

Related links

Add WASI based WebAssembly support #5407

An Update on WebAssembly/WASI Support in Ruby

Production-ready YJIT

YJIT is no longer experimental

Has been tested on production workloads for over a year and proven to be quite stable.

YJIT now supports both x86-64 and arm64/aarch64 CPUs on Linux, MacOS, BSD and other UNIX platforms.

This release brings support for Apple M1/M2, AWS Graviton, Raspberry Pi 4 and more.

Building YJIT now requires Rust 1.58.0+. [Feature #18481]

In order to ensure that CRuby is built with YJIT, please install rustc >= 1.58.0 before running the ./configure script.

Please reach out to the YJIT team should you run into any issues.

The YJIT 3.2 release is faster than 3.1, and has about 1/3 as much memory overhead.

Overall YJIT is 41% faster (geometric mean) than the Ruby interpreter on yjit-bench.

Physical memory for JIT code is lazily allocated. Unlike Ruby 3.1, the RSS of a Ruby process is minimized because virtual memory pages allocated by --yjit-exec-mem-size will not be mapped to physical memory pages until actually utilized by JIT code.

Introduce Code GC that frees all code pages when the memory consumption by JIT code reaches --yjit-exec-mem-size.

RubyVM::YJIT.runtime_stats returns Code GC metrics in addition to existing inline_code_size and outlined_code_size keys: code_gc_count, live_page_count, freed_page_count, and freed_code_size.

Most of the statistics produced by RubyVM::YJIT.runtime_stats are now available in release builds.

Simply run ruby with --yjit-stats to compute and dump stats (incurs some run-time overhead).

YJIT is now optimized to take advantage of object shapes. [Feature #18776]

Take advantage of finer-grained constant invalidation to invalidate less code when defining new constants. [Feature #18589]

The default --yjit-exec-mem-size is changed to 64 (MiB).

The default --yjit-call-threshold is changed to 30.

Regexp improvements against ReDoS

It is known that Regexp matching may take unexpectedly long. If your code attempts to match a possibly inefficient Regexp against an untrusted input, an attacker may exploit it for efficient Denial of Service (so-called Regular expression DoS, or ReDoS).

We have introduced two improvements that significantly mitigate ReDoS.

Improved Regexp matching algorithm

Since Ruby 3.2, Regexp’s matching algorithm has been greatly improved by using a memoization technique.
# This match takes 10 sec. in Ruby 3.1, and 0.003 sec. in Ruby 3.2
/^ab?a$/ =~ "a" * 50000 + "x"
The improved matching algorithm allows most Regexp matching (about 90% in our experiments) to be completed in linear time.

(For preview users: this optimization may consume memory proportional to the input length for each match. We expect no practical problems to arise because this memory allocation is usually delayed, and a normal Regexp match should consume at most 10 times as much memory as the input length. If you run out of memory when matching Regexps in a real-world application, please report it.)

The original proposal is https://bugs.ruby-lang.org/issues/19104

Regexp timeout

The optimization above cannot be applied to some kind of regular expressions, such as those including advanced features (e.g., back-references or look-around), or with a huge fixed number of repetitions. As a fallback measure, a timeout feature for Regexp matches is also introduced.
Regexp.timeout = 1.0
/^ab?a()\1$/ =~ "a" * 50000 + "x"

#=> Regexp::TimeoutError is raised in one second
Note that Regexp.timeout is a global configuration. If you want to use different timeout settings for some special Regexps, you may want to use the timeout keyword for Regexp.new.
Regexp.timeout = 1.0
# This regexp has no timeout

long_time_re = Regexp.new('^ab?a()\1$', timeout: Float::INFINITY)
long_time_re =~ "a" * 50000 + "x" # never interrupted
The original proposal is https://bugs.ruby-lang.org/issues/17837.

Other Notable New Features

SyntaxSuggest
The feature of syntax_suggest (formerly dead_end) is integrated into Ruby. This helps you find the position of errors such as missing or superfluous ends, to get you back on your way faster, such as in the following example:
<div class="language-plaintext highlighter-rouge">
Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
1  class Dog

> 2    defbark

> 3    end

4  end
<p>[<a href="https://bugs.ruby-lang.org/issues/18159">Feature #18159</a>]</p>
ErrorHighlight

Now it points at the relevant argument(s) for TypeError and ArgumentError
test.rb:2:in `+': nil can't be coerced into Integer (TypeError)
sum = ary[0] + ary[1]

^^^^^^
Language
Anonymous rest and keyword rest arguments can now be passed as arguments, instead of just used in method parameters. [Feature #18351]
<div class="language-ruby highlighter-rouge">
  def foo(*)
    bar(*)
  end
  def baz(**)
    quux(**)
  end
A proc that accepts a single positional argument and keywords will no longer autosplat. [Bug #18633]
<div class="language-ruby highlighter-rouge">
proc{|a, **k| a}.call([1, 2])
# Ruby 3.1 and before
# => 1
# Ruby 3.2 and after
# => [1, 2]
Constant assignment evaluation order for constants set on explicit objects has been made consistent with single attribute assignment evaluation order. With this code:
<div class="language-ruby highlighter-rouge">
  foo::BAR = baz
<p><code class="language-plaintext highlighter-rouge">foo</code> is now called before <code class="language-plaintext highlighter-rouge">baz</code>. Similarly, for multiple assignments
to constants, left-to-right evaluation order is used. With this
code:
<div class="language-ruby highlighter-rouge">
    foo1::BAR1, foo2::BAR2 = baz1, baz2
<p>The following evaluation order is now used:</p>

<ol>
  <li><code class="language-plaintext highlighter-rouge">foo1</code></li>
  <li><code class="language-plaintext highlighter-rouge">foo2</code></li>
  <li><code class="language-plaintext highlighter-rouge">baz1</code></li>
  <li><code class="language-plaintext highlighter-rouge">baz2</code></li>
</ol>

<p>[<a href="https://bugs.ruby-lang.org/issues/15928">Bug #15928</a>]</p>
The find pattern is no longer experimental. [Feature #18585]
Methods taking a rest parameter (like *args) and wishing to delegate keyword arguments through foo(*args) must now be marked with ruby2_keywords (if not already the case). In other words, all methods wishing to delegate keyword arguments through *args must now be marked with ruby2_keywords, with no exception. This will make it easier to transition to other ways of delegation once a library can require Ruby 3+. Previously, the ruby2_keywords flag was kept if the receiving method took *args, but this was a bug and an inconsistency. A good technique to find potentially missing ruby2_keywords is to run the test suite, find the last method which must receive keyword arguments for each place where the test suite fails, and use puts nil, caller, nil there. Then check that each method/block on the call chain which must delegate keywords is correctly marked with ruby2_keywords. [Bug #18625] [Bug #16466]
<div class="language-ruby highlighter-rouge">
  def target(**kw)
  end
# Accidentally worked without ruby2_keywords in Ruby 2.7-3.1, ruby2_keywords

# needed in 3.2+. Just like (args, **kwargs) or (...) would be needed on

# both #foo and #bar when migrating away from ruby2_keywords.

ruby2_keywords def bar(args)

target(*args)

end
ruby2_keywords def foo(args)

bar(args)

end
foo(k: 1)
Performance improvements

MJIT

The MJIT compiler is re-implemented in Ruby as ruby_vm/mjit/compiler.

MJIT compiler is executed under a forked Ruby process instead of doing it in a native thread called MJIT worker. [Feature #18968]

As a result, Microsoft Visual Studio (MSWIN) is no longer supported.

MinGW is no longer supported. [Feature #18824]

Rename --mjit-min-calls to --mjit-call-threshold.

Change default --mjit-max-cache back from 10000 to 100.

PubGrub
Bundler 2.4 now uses PubGrub resolver instead of Molinillo.
<ul>
  <li>PubGrub is the next generation solving algorithm used by <code class="language-plaintext highlighter-rouge">pub</code> package manager for the Dart programming language.</li>
  <li>You may get different resolution result after this change. Please report such cases to <a href="https://bounce.depfu.com/github.com/rubygems/rubygems/issues">RubyGems/Bundler issues</a>
RubyGems still uses Molinillo resolver in Ruby 3.2. We plan to replace it with PubGrub in the future.

Other notable changes since 3.1
Data
New core class to represent simple immutable value object. The class is similar to Struct and partially shares an implementation, but has more lean and strict API. [Feature #16122]
    <div class="language-ruby highlighter-rouge">
  Measure = Data.define(:amount, :unit)
  distance = Measure.new(100, 'km')            #=> #<data Measure amount=100, unit="km">
  weight = Measure.new(amount: 50, unit: 'kg') #=> #<data Measure amount=50, unit="kg">
  weight.with(amount: 40)                      #=> #<data Measure amount=40, unit="kg">
  weight.amount                                #=> 50
  weight.amount = 40                           #=> NoMethodError: undefined method `amount='
Hash

Hash#shift now always returns nil if the hash is empty, instead of returning the default value or calling the default proc. [Bug #16908]

MatchData

MatchData#byteoffset has been added. [Feature #13110]

Module

Module.used_refinements has been added. [Feature #14332]

Module#refinements has been added. [Feature #12737]

Module#const_added has been added. [Feature #17881]

Proc

Proc#dup returns an instance of subclass. [Bug #17545]

Proc#parameters now accepts lambda keyword. [Feature #15357]

Refinement

Refinement#refined_class has been added. [Feature #12737]
RubyVM::AbstractSyntaxTree
Add error_tolerant option for parse, parse_file and of. [Feature #19013] With this option

SyntaxError is suppressed

AST is returned for invalid input

end is complemented when a parser reaches to the end of input but end is insufficient

end is treated as keyword based on indent
    <div class="language-ruby highlighter-rouge">
  # Without error_tolerant option
  root = RubyVM::AbstractSyntaxTree.parse(<<~RUBY)
  def m
    a = 10
    if
  end
  RUBY
  # => <internal:ast>:33:in `parse': syntax error, unexpected `end' (SyntaxError)
# With error_tolerant option

root = RubyVM::AbstractSyntaxTree.parse(<<~RUBY, error_tolerant: true)

def m

a = 10

if

end

  RUBY

p root # => #<RubyVM::AbstractSyntaxTree::Node:SCOPE@1:0-4:3>
# end is treated as keyword based on indent

root = RubyVM::AbstractSyntaxTree.parse(<<~RUBY, error_tolerant: true)

module Z

class Foo

foo.

end
def bar
end

end

  RUBY

p root.children[-1].children[-1].children[-1].children[-2..-1]

# => [#<RubyVM::AbstractSyntaxTree::Node:CLASS@2:2-4:5>, #<RubyVM::AbstractSyntaxTree::Node:DEFN@6:2-7:5>]
Add keep_tokens option for parse, parse_file and of. [Feature #19070]
    <div class="language-ruby highlighter-rouge">
  root = RubyVM::AbstractSyntaxTree.parse("x = 1 + 2", keep_tokens: true)
  root.tokens # => [[0, :tIDENTIFIER, "x", [1, 0, 1, 1]], [1, :tSP, " ", [1, 1, 1, 2]], ...]
  root.tokens.map{_1[2]}.join # => "x = 1 + 2"
Set

Set is now available as a builtin class without the need for require "set". [Feature #16989] It is currently autoloaded via the Set constant or a call to Enumerable#to_set.

String

String#byteindex and String#byterindex have been added. [Feature #13110]

Update Unicode to Version 15.0.0 and Emoji Version 15.0. [Feature #18639] (also applies to Regexp)

String#bytesplice has been added. [Feature #18598]
Struct
A Struct class can also be initialized with keyword arguments without keyword_init: true on Struct.new [Feature #16806]
    <div class="language-ruby highlighter-rouge">
  Post = Struct.new(:id, :name)
  Post.new(1, "hello") #=> #<struct Post id=1, name="hello">
  # From Ruby 3.2, the following code also works without keyword_init: true.
  Post.new(id: 1, name: "hello") #=> #<struct Post id=1, name="hello">
Compatibility issues

Note: Excluding feature bug fixes.

Removed constants

The following deprecated constants are removed.

Fixnum and Bignum [Feature #12005]

Random::DEFAULT [Feature #17351]

Struct::Group

Struct::Passwd

Removed methods

The following deprecated methods are removed.

Dir.exists? [Feature #17391]

File.exists? [Feature #17391]

Kernel#=~ [Feature #15231]

Kernel#taint, Kernel#untaint, Kernel#tainted? [Feature #16131]

Kernel#trust, Kernel#untrust, Kernel#untrusted? [Feature #16131]

Stdlib compatibility issues

No longer bundle 3rd party sources
We no longer bundle 3rd party sources like libyaml, libffi.
<ul>
  <li>
    <p>libyaml source has been removed from psych. You may need to install <code class="language-plaintext highlighter-rouge">libyaml-dev</code> with Ubuntu/Debian platform. The package name is different for each platform.</p>
  </li>
  <li>
    <p>Bundled libffi source is also removed from <code class="language-plaintext highlighter-rouge">fiddle</code></p>
  </li>
</ul>
Psych and fiddle supported static builds with specific versions of libyaml and libffi sources. You can build psych with libyaml-0.2.5 like this:
<div class="language-bash highlighter-rouge">
  $ ./configure --with-libyaml-source-dir=/path/to/libyaml-0.2.5
<p>And you can build fiddle with libffi-3.4.4 like this:</p>

<div class="language-bash highlighter-rouge">
  $ ./configure --with-libffi-source-dir=/path/to/libffi-3.4.4
<p>[<a href="https://bugs.ruby-lang.org/issues/18571">Feature #18571</a>]</p>
C API updates

Updated C APIs

The following APIs are updated.

PRNG update

rb_random_interface_t updated and versioned. Extension libraries which use this interface and built for older versions. Also init_int32 function needs to be defined.

Removed C APIs

The following deprecated APIs are removed.

rb_cData variable.

“taintedness” and “trustedness” functions. [Feature #16131]

Standard library updates
Bundler
<ul>
  <li>Add –ext=rust support to bundle gem for creating simple gems with Rust extensions.
[GH-rubygems-6149]
Make cloning git repos faster [GH-rubygems-4475]
RubyGems
<ul>
  <li>Add mswin support for cargo builder. [<a href="https://bounce.depfu.com/github.com/rubygems/rubygems/pull/6167">GH-rubygems-6167</a>]</li>
</ul>
ERB
<ul>
  <li>
ERB::Util.html_escape is made faster than CGI.escapeHTML.

It no longer allocates a String object when no character needs to be escaped.

It skips calling #to_s method when an argument is already a String.

ERB::Escape.html_escape is added as an alias to ERB::Util.html_escape,
which has not been monkey-patched by Rails.
IRB
<ul>
  <li>debug.gem integration commands have been added: <code class="language-plaintext highlighter-rouge">debug</code>, <code class="language-plaintext highlighter-rouge">break</code>, <code class="language-plaintext highlighter-rouge">catch</code>,
next, delete, step, continue, finish, backtrace, info

They work even if you don’t have gem "debug" in your Gemfile.

See also: What’s new in Ruby 3.2’s IRB?
More Pry-like commands and features have been added.

edit and show_cmds (like Pry’s help) are added.

ls takes -g or -G option to filter out outputs.

show_source is aliased from $ and accepts unquoted inputs.

whereami is aliased from @.
The following default gems are updated.
<ul>
  <li>RubyGems 3.4.1</li>
  <li>abbrev 0.1.1</li>
  <li>benchmark 0.2.1</li>
  <li>bigdecimal 3.1.3</li>
  <li>bundler 2.4.1</li>
  <li>cgi 0.3.6</li>
  <li>csv 3.2.6</li>
  <li>date 3.3.3</li>
  <li>delegate 0.3.0</li>
  <li>did_you_mean 1.6.3</li>
  <li>digest 3.1.1</li>
  <li>drb 2.1.1</li>
  <li>english 0.7.2</li>
  <li>erb 4.0.2</li>
  <li>error_highlight 0.5.1</li>
  <li>etc 1.4.2</li>
  <li>fcntl 1.0.2</li>
  <li>fiddle 1.1.1</li>
  <li>fileutils 1.7.0</li>
  <li>forwardable 1.3.3</li>
  <li>getoptlong 0.2.0</li>
  <li>io-console 0.6.0</li>
  <li>io-nonblock 0.2.0</li>
  <li>io-wait 0.3.0</li>
  <li>ipaddr 1.2.5</li>
  <li>irb 1.6.2</li>
  <li>json 2.6.3</li>
  <li>logger 1.5.3</li>
  <li>mutex_m 0.1.2</li>
  <li>net-http 0.3.2</li>
  <li>net-protocol 0.2.1</li>
  <li>nkf 0.1.2</li>
  <li>open-uri 0.3.0</li>
  <li>open3 0.1.2</li>
  <li>openssl 3.1.0</li>
  <li>optparse 0.3.1</li>
  <li>ostruct 0.5.5</li>
  <li>pathname 0.2.1</li>
  <li>pp 0.4.0</li>
  <li>pstore 0.1.2</li>
  <li>psych 5.0.1</li>
  <li>racc 1.6.2</li>
  <li>rdoc 6.5.0</li>
  <li>readline-ext 0.1.5</li>
  <li>reline 0.3.2</li>
  <li>resolv 0.2.2</li>
  <li>resolv-replace 0.1.1</li>
  <li>securerandom 0.2.2</li>
  <li>set 1.0.3</li>
  <li>stringio 3.0.4</li>
  <li>strscan 3.0.5</li>
  <li>syntax_suggest 1.0.2</li>
  <li>syslog 0.1.1</li>
  <li>tempfile 0.1.3</li>
  <li>time 0.2.1</li>
  <li>timeout 0.3.1</li>
  <li>tmpdir 0.1.3</li>
  <li>tsort 0.1.1</li>
  <li>un 0.2.1</li>
  <li>uri 0.12.0</li>
  <li>weakref 0.1.2</li>
  <li>win32ole 1.8.9</li>
  <li>yaml 0.2.1</li>
  <li>zlib 3.0.0</li>
</ul>
The following bundled gems are updated.
<ul>
  <li>minitest 5.16.3</li>
  <li>power_assert 2.0.3</li>
  <li>test-unit 3.5.7</li>
  <li>net-ftp 0.2.0</li>
  <li>net-imap 0.3.3</li>
  <li>net-pop 0.1.2</li>
  <li>net-smtp 0.3.3</li>
  <li>rbs 2.8.2</li>
  <li>typeprof 0.21.3</li>
  <li>debug 1.7.1</li>
</ul>
See GitHub releases like GitHub Releases of logger or changelog for details of the default gems or bundled gems.

See NEWS or commit logs for more details.

With those changes, 3048 files changed, 218253 insertions(+), 131067 deletions(-) since Ruby 3.1.0!

Merry Christmas, Happy Holidays, and enjoy programming with Ruby 3.2!

All Depfu comment commands

@depfu refresh
Rebases against your default branch and redoes this update

@depfu recreate
Recreates this PR, overwriting any edits that you've made to it

@depfu merge
Merges this PR once your tests are passing and conflicts are resolved

@depfu close
Closes this PR and deletes the branch

@depfu reopen
Restores the branch and reopens this PR (if it's closed)

@depfu pause
Pauses all engine updates and closes this PR

depfu · 2024-01-20T07:18:59Z

Closed in favor of #780.

Update Ruby to version 3.2.0

9945591

depfu bot added the depfu label Dec 27, 2022

depfu bot closed this Jan 20, 2024

depfu bot deleted the depfu/engine/ruby-3.2.0 branch January 20, 2024 07:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ruby] Upgrade Ruby to 3.2.0 #740

[ruby] Upgrade Ruby to 3.2.0 #740

depfu bot commented Dec 27, 2022

depfu bot commented Jan 20, 2024

[ruby] Upgrade Ruby to 3.2.0 #740

[ruby] Upgrade Ruby to 3.2.0 #740

Conversation

depfu bot commented Dec 27, 2022

What changed?

3.2.0

WASI based WebAssembly support

Background

Use case

Technical points

Related links

Production-ready YJIT

Regexp improvements against ReDoS

Improved Regexp matching algorithm

Regexp timeout

Other Notable New Features

SyntaxSuggest

ErrorHighlight

Language

Performance improvements

MJIT

PubGrub

Other notable changes since 3.1

Compatibility issues

Removed constants

Removed methods

Stdlib compatibility issues

No longer bundle 3rd party sources

C API updates

Updated C APIs

Removed C APIs

Standard library updates

depfu bot commented Jan 20, 2024