Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC submission for override_gem_dependency_versions #13

Closed
wants to merge 3 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions text/0000-override-gem-dependency-versions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
- Feature Name: override_gem_dependency_version
- Start Date: 2018-07-10
- RFC PR:
- Bundler Issue:

# Summary

Enable the version constraints of gem dependencies to be explicitly overridden for gems listed in a Gemfile.

The purpose of this RFC is to get feedback on use cases, format/syntax/DSL, and feature options.

# Motivation

This feature has been requested many times since 2011.

Project authors want the ability to specify newer versions of gems in their Gemfile even if the version is being constrained by another gem.

There several causes for this issue:
- The version dependency specified in the gemspec is unnecessarily strict: `s.add_dependency 'json', '= 1.7.7'`
- The gem is slow to update or is no-longer maintained to the point that version constraints do not keep up with the version updates of their dependencies: `s.add_dependency 'json', '>= 1.7.7', '<= 2'`

The common approach to this problem is to fork the problematic gem, relax the dependencies specified in the gemspec, and reference the fork in the Gemfile. This is a very heavyweight and time intensive solution to a problem that could be easily solved by a well-designed solution to override dependencies from within the Gemfile itself.

Obviously, overriding version constraints of gems would be *AT YOUR OWN RISK*, however, a properly defined DSL and output messages should mitigate and address these issues.

Most of the time, this kind of override should *NOT* be a permanent solution, and as such, *SHOULD* be accompanied with some type of message that is additionally displayed to indicate why the override has occurred.

# Guide-level explanation

Say you want to use the latest version of the `json` gem (currently `2.1.0`), however, another gem you are using constrains the version to `s.add_dependency 'json', '>= 1.7.7', '<= 2'`. To override this version constraint so that version `2.1.0` is installed:

```ruby
gem 'json'
gem 'other_gem' do |s|
s.override_runtime_dependency('json', '>= 1.7.7', '<3')
end
```

This will allow bundler to install the `2.1.0` version of the `json` gem.

The output when running `bundle install` will look like the following:

```
Installing json 2.1.0 (forced version, conflicts with other_gem)
Installing other_gem 1.0.0
```

Most of the time, this should only be a temporary fix as we expect the gem author to release a new version with updated dependency constraints.

To provide additional information a note can be included:

```ruby
gem 'json'
gem 'other_gem' do |s|
s.override_runtime_dependency('json', '>= 1.7.7', '<3', 'waiting on pull request #5')
end
```

The output when running `bundle install` will look like the following:

```
Installing json 2.1.0 (forced version, conflicts with other_gem: waiting on pull request #5)
Installing other_gem 1.0.0
```

# Reference-level explanation (TODO)

This is the technical portion of the RFC. Explain the design in sufficient detail that:

- Its interaction with other features is clear.
- It is reasonably clear how the feature would be implemented.
- Corner cases are dissected by example.

The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.

# Drawbacks

These are the common drawbacks and counter-arguments mentioned by the community:
* This issue can already be solved by forking gem repositories and updating the dependencies yourself
* Allowing this means there is no longer a canonical source of truth about what a gem's dependencies are
* When these kinds of changes cause resolver errors (and they will), it will be extremely hard to diagnose
* The potential that people will complain that Bundler is broken when it’s actually something they did themselves
* The potential that people will complain to the gem author that the gem is broken when it’s actually something they did themselves

These are all valid points of view, however, there are also counter-arguments to each of these points which are also equally valid:
* The most obvious place to manage version dependencies (even overridden version constraints) is in the Gemfile and not in a forked repository
* Forking an entire project (and subsequently managing upstream changes, etc.) is an unnecessarily tall order for something that can certainly be fixed with a simple, at-your-own-risk dependency override
* This feature becomes more and more important with the number of gems growing. .NET has it. Java has it. Nothing bad happened to them because of it
* Maintaining overridden version constraints within the Gemfile saves a significant amount of development cost when compared to maintaining a forked repository
* Explicit syntax within the Gemfile with custom notes and obvious output messages when running `bundle install` can help to mitigate developers not knowing that a version conflict exists

# Rationale and Alternatives
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a good example of the problem in action: salesforce-marketingcloud/FuelSDK-Ruby#115.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, but I don't get it. But what makes it actual good example?

Copy link
Author

@br3nt br3nt Nov 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you prefer to maintain an entire fork of a gem just to update it's Gemfile, when you could just add an override in your own Gemfile?

The good things about overriding in your own Gemfile are:

  • gem install can explicitly show that a problematic gems dependencies are being forced to be a version that isn't technically supported by that gem. This couldn't happen if you maintain a fork obviously. Maybe you can tell it's not the original gem because you can see the link to a GitHub repo, but it doesn't tell people why you are doing this and what the condition is for moving back to the original gem. Maybe you could add a code comment the explains all this, but that's not very semantic. You couldn't use a code tool to read the comments to tell you when you should start using the original gem again.
  • gem install can explicitly show that none of the problematic gem's dependencies' versions are being forced. If after a gem update this message is being show, it indicates that the original gem author has finally updated their problematic gem dependency version constraints, which means you can finally remove the override from your Gemfile. This wouldn't happen if you maintain a fork. You would need to constantly actually look at the original gem for updates. That is such a manual chore.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you prefer to maintain an entire fork of a gem just to update it's Gemfile, when you could just add an override in your own Gemfile?

I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.

Ok, but why?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to maintain an entire fork of a unmaintained gem or actually try to find out alternative to this gem to remove abandoned gem in all codebases I do maintain.

For the cases where that apply to us, we have found maintaining and tracking occasionally updated (but not fully abandoned) gems, ends up being a thankless chore that creates more problems than it solves. This of course doesn't mean it's right for everyone else.

This is also a common feature in other dependency managers:

The above cover the top languages from Github and all have a solution. I would argue this is not an esoteric need.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote up my view on the problem and options for solutions that are currently available, 1 that is this RFC and 1 other that is not available.

I hope this helps to frame the debate.

Options for handling out of date gems

Out of date gems have been a problem for a long time. As the ruby ecosystem evolves and ages, they are and will become more common. This is especially prevalent in areas where the state of the art has moved on (for example SOAP).

Unfortunately, these 'stuck' gems often have dependencies on other more common and evolving gems. This would not be a problem if the un-evolved gems had loose dependency constraints. However, the recommendation for gem writers is to be relatively tight on constraints to ensure compatibility.

We now have stuck gems depending on other gems. Since only one version of a gem can be installed in bundler. This then constrains the use of newer gems that share a downstream gem dependency (but not the dependency version).

There are a number of options for handling these out of date gems. Here I list them out and highlight their pros and cons.

Finally, I give my recommendation for consideration.

Here are the many options that developers have to deal with out of date gems:

  1. Create an issue
  2. Create issue and develop a PR
  3. Monkeypatch
  4. Fork and use git path
  5. Fork and use private registry
  6. Fork and change name and alias name
  7. Take over maintenance of the gem
  8. Force version
  9. Write your own

Here is a detailed description of each of them and their pros and cons:

1. Create an issue

Create an issue on the relevant repo and wait for the maintainer to handle it and release a new version. This and the next option are the happy path, the way things should work.

This option and the following option are usually the first port of call, since they are also the easiest.

Pros

  • Keeps gem up to date for the community
  • Easy for the user

Cons

  • No help if the gem isn't being maintained
  • Have to wait for maintainer to agree to and then make the change a priority
  • Assumes gem is maintained in timely manner
  • Maintainer has to do the work

2. Create issue and submit a PR

Create the issue but also provide a solution in the form of a PR. Again, this is the happy path, but isn't always available.

Pros

  • Keeps gem up to date for the community
  • A ready made solution is in place so the maintainer has less work
  • Relatively easy for some users

Cons

  • No help if the gem isn't being maintained
  • Have to wait for maintainer to make it a priority
  • Assumes gem is maintained in timely manner
  • You have to do the work, and if you don't understand the code, figure it out.

3. Monkeypatch

Create a gem or an internal piece of code that overrides the buggy or out of date behavior in the gem that needs updating. An example of this is Savon Fixes which I created.

Pros

  • Make the change without having to get the maintainers to do anything
  • If a gem, can be provided to the community

Cons

  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Fragile (especially if a gem; a monkeypatch gem is not straightforward)
  • Doesn't work for out of date dependencies as gemspecs can't be monkeypatched
  • Depending on load order, can be very tricky to verify that it is working if deeply embedded in a chain of gems

4. Fork and use git path

This is the most common way of forking. Create the fork, make the patch and point your gem file at your git repo and the relevant commit/branch.

Pros

  • Can make targeted changes of whatever nature is needed

Cons

  • Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
  • Requires discipline. I've seen additional functionality added to the forked gem, thus creating barriers to tracking upstream.
  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Can't have all gems in a registry

5. Fork and use private registry

Create the fork, make the patch, add in your proprietary CI/CD and registry config/code and point your Gemfile at the internal registry.

Pros

  • Can make targeted changes of whatever nature is needed

Cons

  • Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
  • Requires discipline. I've seen additional functionality added to the forked gem, thus creating barriers to tracking upstream.
  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Having to add internal CI config/code to the public repo
  • Having to create new version number to avoid clash with public version (so multiple sources aren't ambiguous)
  • If there are intermediate gems, having to make sure they will accept the new version number

6. Public fork and change name

In this case you are forking the gem, and then changing it's name so that you can publish to Rubygems under the alternate name. Then your Gemfile points to the newly named gem.

Pros

  • Can use standard publishing mechanisms (Rubygems)
  • Community can benefit from changes (if aware)

Cons

  • In order for intermediate gems in a dependency chain to use, would require alias_gem_name fuctionality to Bundler which is not currently available
  • Multiple versions of same gem with similar names will exist
  • Could cause missing out on original gem updates

7. Take over maintenance of the gem

In this case, become the maintainer of the gem as the previous maintainer no longer has the resources to spend the time on maintenance. Then you can make the necessary updates.

Pros

  • Keeps gem up to date for the community
  • Gem is maintained again
  • Strong way to give back to the community

Cons

  • Big task to take on
  • Maintaining a full code base (you have to do the work, and if you don't understand the code, figure it out)
  • When maintainer has disappeared, taking over maintainership is not easy
  • Maintainer may not be willing to give up ownership, even as they no longer make changes. (I have an example of this.)
  • To be a universal approach, requires more robust or frequent 'change of ownership' process for Rubygems.org

8. Force version

Add a capability Bundler which allows an option in a Gemfile. This option lets the user force the version of a dependency which then overrides that dependency in the gemspecs of all dependent gems.

Pros

  • Allows gem dependencies to be forced when the gem is still working (other than the out of date dependencies)
  • Works on dependencies at any level in a dependency chain
  • Javascript (nvm and yarn), Python (pip) and Java (Maven, Gradle), the top 3 Github languages, offer this capability

Cons

  • Could cause missing out on gem updates
  • Community doesn't move forward together because it could cause people to avoid proposing issues and changes to gems (options 1 and 2)
  • Change to Bundler api (though opt in)
  • Does not solve cases where functionality/API changes are required

9. Write your own

This just means avoiding the 'stuck' gem entirely.

Pros

  • Full control

Cons

  • Not feasible in situations where the capabilities are not a strength of yours (in our case, SOAP)
  • Does not solve problems for intermediate gems in a dependency chain
  • If private, no benefit to or from the community
  • If public, have to popularize your version so that all gem maintainers switch

Recommendation

Given the complexity and how painful many of the options are, I believe adding 8) Force Version to Bundler is the best solution for the dependency side of these problems.

Additionally, I think the Bundler team should consider an alias gem name function that would allow users to redirect to a forked version of a gem which they can make public without having to fork entire chains of gems.

References

Dependency managers offering override functionality:


This is the best design because it is very explicit and clearly documents what is happening.
It requires project maintainers to have a clear understanding of the conflicts they are encountering.

Other designs are as follows:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking at the original design by @segiddins and now ported to bundler/master here: rubygems/rubygems#4178, I think the force_version syntax is better. I couldn't really see any advantages to the more complex implementation other than a very specific message. My vote is that force_version provides most of the benefit with a simpler syntax.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is sample output from the force_version PR:

$ dbundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Using public_suffix 4.0.6
Using builder 3.2.4
Using mini_portile2 2.4.0
Using bundler 2.3.0.dev
Using rack 2.2.3
Using socksify 1.7.1
Using json 1.8.6
Using jwt 1.5.6
Using nori 2.6.0
Using addressable 2.7.0
Using gyoku 1.3.1
Using nokogiri 1.10.10
Using httpi 2.4.5
Using akami 1.3.1
Using wasabi 3.6.1
Using savon 2.12.1 [version forced]
Using sfmc-fuelsdk-ruby 1.3.0
Bundle complete! 2 Gemfile dependencies, 17 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

This also means I no longer have to maintain a forked version of sfmc-fuelsdk-ruby.

Here is the Gemfile:

source 'https://rubygems.org'

gem "sfmc-fuelsdk-ruby", "1.3.0"
gem "savon", "2.12.1", force_version: true

Here is the Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.7.0)
      public_suffix (>= 2.0.2, < 5.0)
    akami (1.3.1)
      gyoku (>= 0.4.0)
      nokogiri
    builder (3.2.4)
    gyoku (1.3.1)
      builder (>= 2.1.2)
    httpi (2.4.5)
      rack
      socksify
    json (1.8.6)
    jwt (1.5.6)
    mini_portile2 (2.4.0)
    nokogiri (1.10.10)
      mini_portile2 (~> 2.4.0)
    nori (2.6.0)
    public_suffix (4.0.6)
    rack (2.2.3)
    savon (2.12.1)
      akami (~> 1.2)
      builder (>= 2.1.2)
      gyoku (~> 1.2)
      httpi (~> 2.3)
      nokogiri (>= 1.8.1)
      nori (~> 2.4)
      wasabi (~> 3.4)
    sfmc-fuelsdk-ruby (1.3.0)
      json (~> 1.8, >= 1.8.1)
      jwt (~> 1.0, >= 1.0.0)
      savon (= 2.2.0)
    socksify (1.7.1)
    wasabi (3.6.1)
      addressable
      httpi (~> 2.0)
      nokogiri (>= 1.4.2)

PLATFORMS
  x86_64-darwin-19

DEPENDENCIES
  savon (= 2.12.1)
  sfmc-fuelsdk-ruby (= 1.3.0)

BUNDLED WITH
   2.3.0.dev

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My implementation in rubygems/rubygems#4178 is now fulling working. This is a port of @segiddins original POC. I have created additional tests, as well as run it against our complex rails project at http://https://github.com/simplybusiness where it has worked well. @br3nt, would welcome your thoughts.

This feature would have a lot of value for us as a company. It is also opt in so no one is :forced to use it ;-).


```ruby
gem "puma", "5.1" # depends on rack <= 2.0
gem "rack", "3.0", force_version: true
```

This version forces the `rack` gem to explicitly force the install of a version matching `3.0`.
This syntax will work fine, but it is not documenting which is the problematic gem preventing the requested version of `rack`.

The equivalent using the proposed solution would look like this:

```ruby
gem "puma", "5.1" do |s|
s.override_runtime_dependency "rack", "~> 3.0"
end
gem "rack", "3.0"
```

The proposed solution clearly documents that the problematic gem is `puma` and that we are explicitly overriding it's version constraints for the `rack` gem.

The impact of this is verbose and possibly unsightly code. The benefit, however, is that version overrides are explicitly documented which will help prevent excessive issues being lodged to bundler and to gem authors.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it also could make the implementation of the feature much more difficult


# Unresolved questions

* What is the best mechanism to show override messages to the user?
* What is the best syntax to include a custom override message?