Merge pull request #335 from garrettrowell/vendored_modules

Add support for downloading *_core modules

.rubocop_todo.yml

@@ -1,6 +1,6 @@
 # This configuration was generated by
 # `rubocop --auto-gen-config`
-# on 2024-11-14 21:32:23 UTC using RuboCop version 1.67.0.
+# on 2024-11-14 22:37:24 UTC using RuboCop version 1.67.0.
 # The point is for the user to remove these configuration records
 # one by one as the offenses are removed from the code base.
 # Note that changes in the inspected code, or installation of new
@@ -22,13 +22,6 @@ Gemspec/AddRuntimeDependency:
   Exclude:
     - 'onceover.gemspec'
 
-# Offense count: 1
-# Configuration parameters: Severity, Include.
-# Include: **/*.gemspec
-Gemspec/RequiredRubyVersion:
-  Exclude:
-    - 'onceover.gemspec'
-
 # Offense count: 2
 # This cop supports safe autocorrection (--autocorrect).
 # Configuration parameters: EnforcedStyleAlignWith.
@@ -146,12 +139,13 @@ Layout/HeredocIndentation:
     - 'lib/onceover/cli/show.rb'
     - 'lib/onceover/cli/update.rb'
 
-# Offense count: 7
+# Offense count: 8
 # This cop supports safe autocorrection (--autocorrect).
 # Configuration parameters: Width, AllowedPatterns.
 Layout/IndentationWidth:
   Exclude:
     - 'lib/onceover/controlrepo.rb'
+    - 'lib/onceover/vendored_modules.rb'
 
 # Offense count: 12
 # This cop supports safe autocorrection (--autocorrect).
@@ -287,14 +281,13 @@ Layout/TrailingEmptyLines:
     - 'features/step_definitions/formatter.rb'
     - 'features/support/env.rb'
 
-# Offense count: 9
+# Offense count: 8
 # This cop supports safe autocorrection (--autocorrect).
 # Configuration parameters: AllowInHeredoc.
 Layout/TrailingWhitespace:
   Exclude:
     - 'features/step_definitions/common.rb'
     - 'features/step_definitions/formatter.rb'
-    - 'lib/onceover/deploy.rb'
     - 'lib/onceover/node.rb'
     - 'lib/onceover/rspec/formatters.rb'
     - 'lib/onceover/testconfig.rb'
@@ -544,7 +537,7 @@ Style/ConditionalAssignment:
     - 'lib/onceover/test.rb'
     - 'lib/onceover/testconfig.rb'
 
-# Offense count: 26
+# Offense count: 27
 # Configuration parameters: AllowedConstants.
 Style/Documentation:
   Enabled: false
@@ -578,7 +571,7 @@ Style/Encoding:
   Exclude:
     - 'onceover.gemspec'
 
-# Offense count: 33
+# Offense count: 34
 # This cop supports unsafe autocorrection (--autocorrect-all).
 # Configuration parameters: EnforcedStyle.
 # SupportedStyles: always, always_true, never
@@ -943,7 +936,7 @@ Style/WhileUntilDo:
   Exclude:
     - 'lib/onceover/controlrepo.rb'
 
-# Offense count: 24
+# Offense count: 32
 # This cop supports safe autocorrection (--autocorrect).
 # Configuration parameters: AllowHeredoc, AllowURI, URISchemes, IgnoreCopDirectives, AllowedPatterns.
 # URISchemes: http, https

README.md

@@ -24,6 +24,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others
     - [Puppetfile](#puppetfile)
   - [Spec testing](#spec-testing)
     - [Adding your own spec tests](#adding-your-own-spec-tests)
+    - [Vendored Modules](#vendored-modules)
   - [Using Workarounds](#using-workarounds)
   - [Extra tooling](#extra-tooling)
     - [Plugins](#plugins)
@@ -36,6 +37,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others
     - [Ruby Warnings](#ruby-warnings)
     - [Rake tasks](#rake-tasks)
       - [generate_fixtures](#generate_fixtures)
+      - [generate_vendor_cache](#generate_vendor_cache)
 
 ## Overview
 
@@ -616,6 +618,30 @@ If you want to see Puppet's output, you can set the `SHOW_PUPPET_OUTPUT` environ
 
 `SHOW_PUPPET_OUTPUT=true onceover run spec`
 
+### Vendored Modules
+
+As of Puppet 6.0 some resource types were removed from Puppet and repackaged as individual modules. These supported type modules are still included in the `puppet-agent` package, so you don't have to download them from the Forge. However, this does not apply to the `puppet` gem used when spec testing. This frequently results in users wondering why their Puppet manifests apply just fine on a node, but their tests fail with messages like `Unknown resource type: cron_core` for example. A common workaround for this problem was to add said modules into your Puppetfile, thus requiring manual management.
+
+Onceover now has the ability to remove that manual process for you by querying Github's API to determine which versions are in use by the version of the [puppet-agent package](https://github.com/puppetlabs/puppet-agent/tree/main/configs/components) you are testing against.
+
+This functionality is opt in, so to use it configure the following:
+
+```yaml
+# onceover.yaml
+opts:
+  auto_vendored: true
+```
+
+or on the cli:
+
+```shell
+bundle exec onceover run spec --auto_vendored=true
+```
+
+Essentially what this is doing is resolving any of these [supported type modules](https://www.puppet.com/docs/puppet/8/type#supported-type-modules-in-puppet-agent) that are not already specified in your Puppetfile, and adding them to the copy Onceover uses to deploy into its working directory structure.
+
+CI/CD pipeline users are encouraged to provide Onceover with a cache of the module versions to test against in order to avoid hitting Githubs API ratelimit. To do so, the [generate_vendor_cache](#generate_vendor_cache) rake task can be used to populate the cache into your `spec/vendored_modules` directory.
+
 ## Using workarounds
 
 There may be situations where you cannot test everything that is in your puppet code, some common reasons for this include:
@@ -890,6 +916,12 @@ fixtures:
 
 Notice that the symlinks are not the ones that we provided in `environment.conf`? This is because the rake task will go into each of directories, find the modules and create a symlink for each of them (This is what rspec expects).
 
+#### generate_vendor_cache
+
+`bundle exec rake generate_vendor_cache`
+
+This task will query Github's API to determine the versions of the vendored modules in use by the version of the puppet agent you are testing against, and cache that information in `control-repo/spec/vendored_modules`. This way your pipelines won't need to reach out for this information each time Onceover is ran with `auto_vendored` enabled.
+
 ## Developing Onceover
 
 Install gem dependencies:

features/auto_vendored.feature

@@ -0,0 +1,27 @@
+@vendored @puppet6
+Feature: Automatically resolve modules vendored with puppet-agent package
+  Onceover should optionally attempt to resolve these vendored modules so that
+  users do not need to maintain these in their Puppetfile's unless they have a reason
+  to do so
+
+  Background:
+    Given onceover executable
+
+  Scenario: Auto resolve disabled and Puppetfile empty
+    Given initialized control repo "vendored"
+    When I run onceover command "run spec" with class "role::cron"
+    Then I should see error with message pattern "Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'cron'"
+
+  Scenario: Auto resolve enabled and Puppetfile empty
+    Given existing control repo "vendored"
+    When I run onceover command "run spec --auto_vendored=true" with class "role::cron"
+    Then the temporary Puppetfile should contain /mod 'puppetlabs-cron_core',\n.*git: 'https://github.com/puppetlabs\/puppetlabs-cron_core.git',\n.*ref: 'refs\/tags\/.*'/
+    And I should not see any errors
+
+  Scenario: Auto resolve enabled and cron_core specified in Puppetfile
+    Given existing control repo "vendored"
+    When I run onceover command "run spec --auto_vendored=true" with --puppetfile Puppetfile.cron
+    Then I should see message pattern "cron_core found in Puppetfile. Using the specified version"
+    Then the temporary Puppetfile should contain /mod 'puppetlabs\/cron_core'/
+    And I should not see any errors
+

lib/onceover/cli/run.rb

@@ -44,6 +44,7 @@ def self.command
             optional nil, :format, 'Which RSpec formatter to use, valid options are: documentation, progress, FailureCollector, OnceoverFormatter. You also specify this multiple times', multiple: true, default: :defaults
             optional nil, :no_workarounds, 'Disables workarounds that have been added for convenience to get around common RSPec issues such as https://github.com/rodjek/rspec-puppet/issues/665'
             optional :ff, :fail_fast, 'Abort the run after the first failure'
+            optional nil, :auto_vendored, 'Attempt to resolve vendored puppet modules. Ex: puppetlabs/cron_core', default: false
 
             run do |opts, args, cmd|
               repo = Onceover::Controlrepo.new(opts)

lib/onceover/deploy.rb

@@ -13,6 +13,10 @@ def deploy_local(repo = Onceover::Controlrepo.new, opts = {})
       skip_r10k_default = !(File.file?(repo.puppetfile))
       skip_r10k = opts[:skip_r10k] || skip_r10k_default
       force     = opts[:force] || false
+      # Only attempt to resolve vendored modules if configured to do so
+      auto_vendored = opts[:auto_vendored] || false
+
+      require 'onceover/vendored_modules' if auto_vendored
 
       if repo.tempdir == nil
         repo.tempdir = Dir.mktmpdir('r10k')
@@ -90,7 +94,26 @@ def deploy_local(repo = Onceover::Controlrepo.new, opts = {})
         if /:control_branch/.match(puppetfile_contents)
           logger.debug "replacing :control_branch mentions in the Puppetfile with #{git_branch}"
           new_puppetfile_contents = puppetfile_contents.gsub(":control_branch", "'#{git_branch}'")
-          File.write("#{temp_controlrepo}/Puppetfile", new_puppetfile_contents)  
+          File.write("#{temp_controlrepo}/Puppetfile", new_puppetfile_contents)
+        end
+
+        if auto_vendored
+          tmp_puppetfile = File.join(temp_controlrepo, 'Puppetfile')
+          tmp_puppetfile_contents = File.read(tmp_puppetfile)
+          vm = Onceover::VendoredModules.new({ repo: repo })
+          puppetfile = R10K::ModuleLoader::Puppetfile.new(basedir: temp_controlrepo)
+          vm.puppetfile_missing_vendored(puppetfile)
+          unless vm.missing_vendored.empty?
+            missing_slugs = vm.missing_vendored.map do |missing_mod|
+              missing_mod.keys[0]
+            end
+            logger.debug "Adding #{missing_slugs} to #{tmp_puppetfile}"
+            modlines = vm.missing_vendored.map do |missing_mod|
+              mod_slug = missing_mod.keys[0]
+              "mod '#{mod_slug}',\n  git: '#{missing_mod[mod_slug][:git]}',\n  ref: '#{missing_mod[mod_slug][:ref]}'"
+            end.join("\n")
+            File.write(tmp_puppetfile, "#{tmp_puppetfile_contents}\n# Onceover Managed Vendored Modules\n#{modlines}")
+          end
         end
       end
 

lib/onceover/rake_tasks.rb

@@ -88,3 +88,12 @@
   end
 
 end
+
+desc 'Create cache for vendored modules'
+task :generate_vendor_cache do
+  require 'onceover/controlrepo'
+  require 'onceover/vendored_modules'
+
+  repo = Onceover::Controlrepo.new(debug: true)
+  Onceover::VendoredModules.new({ repo: repo, cachedir: File.join(repo.spec_dir, 'vendored_modules'), force_update: true })
+end