Transitive github overrides

[lamont-granquist]

We need to solve the problem where a base cookbook like build-essentials needs to be overridden with a cookbook out of git. There are two possible use cases for this. One is that the user is forking the base cookbook in their organization due to some kind of specialization at their site (which might be bugs that they're fixing or might be site-specific features that they want to submit upstream). There is also a use case where you're the cookbook author and you want to bake a cookbook in your own site for awhile before releasing it upstream. There is also the use case of bugfixing before releasing it upstream.

When testing cookbooks the existing behavior creates a problem because the github override needs to be present at every 'testable' level of cookbooks. All the leaf level application cookbooks need to have the override, its useful to test the base role in isolation and its often useful to have interior cookbooks which roll up what it is to be "a rails application in the org" or "a monitored host" and test those in isolation as well. The solution of rolling up the dependencies under an intermediate role or role cookbook which all applications use a dependency and overriding them there is precisely what is broken by not having any transitive git dependencies -- I want a way to say that "all of my rails apps pull this cookbook from git" and be able to specify that in a single place.

There is the possible solution of running your own supermarket site but that is a very high barrier to entry when all you want to do is patch build-essentials and move on with your life.

There is also the possible solution of using your own chef-server, but that requires a chef-server to act as a berks API server and requires uploading the development cookbook to the chef-server simply in order to test.

There are problems in simply following dependencies through github. One is the 'finished artifact' problem where the metadata.rb on github has not been compiled. This is a problem for only a segment of the chef population which is using libraries to generate version from CI. The other problem is that depsolving through github cookbooks becomes a long, iterative process. The current algorithm for depsolving is that it can grab the universe and depsolve. If we honor transitive PolicyFiles or PolicyFile.locks then we will need to grab github cookbooks and then re-depsolve with the new contraints and then potentially grab more cookbooks from github and depsolve again, etc.

A solution which allows site-overrides and avoids transitive following of PolicyFiles would be to allow mixing in a global PolicyFile which "dependency injects" a solution to point a cookbook at github (or at any other source). Since this solution supports equality pinning (or any other version pinning) then it should solve the use cases that drive the desire for "transitive PolicyFile.lock" behavior as well.

[lamont-granquist]

One crazy idea that just popped into my head was to allow roles and environments to not only have cookbook dep contraints, but to have sources as well. Since we bake roles and environments and the PolicyFile into the PolicyFile.lock there's a way right there to do a 'mixin' or 'include' that we're already doing, we're just not doing it for cookbook sources.

[lamont-granquist]

So we've got this reduced down to this now when using a monorepo:

% cat PolicyFile.rb
name "sk_role_base"
run_list "sk_role_base::default"
default_source :supermarket
default_source :chef_repo, ".." do |s|
  s.preferred_for "multipackage", "sk_ruby", "sk_s3_file", "sk_ssh_known_hosts", "sk_sysctl"
end
cookbook "sk_role_base", path: "."

All we need to do is to add something like:

source_priority_list [ :chef_repo, :supermarket ]

Along with #1086 and we can get that down to:

% cat PolicyFile.rb
name "sk_role_base"
run_list "sk_role_base::default"
default_source :supermarket
default_source :chef_repo
source_priority_list [ :chef_repo, :supermarket ]
metadata

which would be pretty much boilerplate plus the name of the cookbook.

doesn't really solve transitive github stuff, but it externalizes that to the chef_repo. that could be a big git monorepo or you could do crazy git submodule shit, but it externalizes the problem of synch'ing the state of your own git universe. there's likely a problem here with getting devs to keep the repo up-to-date before doing policyfile operations.

[afiune]

@lamont-granquist while doing some grooming we have detected that this issue is 5 years old, from your point of view, is this still a valid idea or could we proceed to close it?