Reconsider the approach to Bundler? (Stop removing BUNDLED WITH from lockfile)

[dentarg]

Replying to #1294 (comment) but in a new issue as the old one has been closed now and I think this topic deserves its own thread.

That's one of the reasons why (i suspect) some of you report upgrading bundler (via my patch/branch working) as either it has a bugfix for this problem or the bundler version that ships with 3.1.1 had a bug in it.

Yes, the problem we all ran into was rubygems/rubygems#5351, a regression introduced in Bundler 2.3.7. Ruby 3.1.1, released Feb, 18 vendors Bundler 2.3.7. The bug was fixed with rubygems/rubygems#5386 and Bundler 2.3.9 (released March, 9).

It's important to note that even though we say "using bundler 2.2.x" in your Heroku output, due to the way bundler works, a vendored version of bundler that is higher than our version will be used.

Bundler 2.3.0 introduced Change bundle install with a lockfile to respect the BUNDLED WITH bundler version (rubygems/rubygems#4076) but as Heroku removes the BUNDLED WITH section, that is not used on Heroku.

@schneems are you willing to consider changing the buildpack to not remove the BUNDLED WITH section? Could it be done as an opt-in somehow?

I know there are extensive documentation https://devcenter.heroku.com/articles/ruby-support#libraries, https://devcenter.heroku.com/articles/bundler-version around Bundler and the various issues we have all encountered in the past, but maybe it is time to consider alternatives for the future? Bundler has evolved quite a bit since v1.x.

[schneems]

In the CNB we install that exact version of bundler captured in the Gemfile.lock https://github.com/heroku/buildpacks-ruby

It might make sense to backport that behavior here and deprecate the current "blessed version" system.

Bundler has evolved quite a bit since v1.x.

A bit pithy, but...that rapid evolution without accompanying major version revisions is part of the problem. The 1.x to 2.0 transition was extremely rocky. Bundler keeps being more intertwined with the Ruby ecosystem and this behavior is harder to avoid. For example in the preview and RC releases it uses whatever version comes with Ruby because it's newer than what we provide. So really we're fighting the tide here.

This move away from specific, known good versions will be a stability hit for customers (as often they're installing in production with different flags than locally, and likely with different cached gems too) and it also removes a support vector but it seems like the right direction to go.

Some of the more extreme bundler features like re-exec-ing itself were part of the reason I didn't want to allow arbitrary bundler versions as these features took a while to stabilize (not to mention features are added in non-minor releases). I think they're relatively baked now and overall using the exact same bundler version you use locally seems like a generalized win.

No guarantees on timeline, but letting you know where I'm at.