catalog access / catalog transformations

[DavidS]

Use Case

• security assertions on the catalog
• global transmutations of the catalog
• generating component resources from native providers (e.g. concat -> file)
• agent-side catalog operations (e.g. deferred functions)

Describe the Solution You Would Like

Add new compilation stages before the catalog is transmitted from the server and after it has been received at the agent. Allow modules to inject transformation functions that can mutate the catalog before further processing. Each of the transformation functions would take (part of) the catalog and be able to add/change/delete resources as it sees fit.

To help operators understand the impact of transformation functions, they should provide metadata of which types the function looks at and which types get emitted by them. This information can be used to ease implementation (only pass in requested resources, reducing filtering requirements) and safety (check that only expected types are returned) of those functions. At the same time this information can be used to ascertain a safe application order of the transforms:

• two or more transforms requesting different type inputs are always safe to run and do not require a specific order
  • given f(A): B and g(C): D, f and g have no order requirement
• two or more transforms requesting the same input types are always safe to run and do not require a specific order
  • given f(A): B and g(A): C, f and g have no order requirement
• a transform has to run after all other transforms that output required types for this transform
  • given f(A): B and g(B): C, f needs to be applied before g
• a cycle in transform type requirements is illegal
  • given f(A): B and g(B): A, f and g are incompatible
  • except for the special case of a single transform f(A): A, which can be safely applied

Should this turn out to be too restrictive for useful configurations, we can consider allowing folks to provide a partial priority override for transforms that conflict. Caveat emptor that in those cases the operator needs to take on additional responsibilities to understand and sequence those transforms and the consequences that'll have on the catalog.

Describe Alternatives You've Considered

Exposing the current mechanisms for doing these kinds of transformations would make implementing the Resource API outside of puppet a lot harder, while at the same time perpetuating the brittleness and opaqueness of the generate approach.

Additional Context

Originally tracked in https://tickets.puppetlabs.com/browse/PDK-533 , https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource-api/README.md#catalog-access

[DavidS]

Other examples:

• https://github.com/voxpupuli/puppet-firewalld/blob/e23f7688c6d045c17dc7fe4dd09982a1c64cbf65/lib/puppet/type/firewalld_zone.rb#L39
• https://github.com/simp/pupmod-simp-iptables/blob/f2c5fd136c1dfa2f5fb4e074cab2965931f0b512/lib/puppet/provider/iptables_optimize/optimize.rb
• https://github.com/simp/pupmod-simp-deferred_resources
  • Create if it doesn't exist in the catalog, no error otherwise
  • In some cases, modify attributes of existing resources
  • Basically, it's an exception-catching ensure_resources combined with a resource collector that doesn't break if something doesn't exist in the catalog
• @seanmil on slack writes:

  my use case is that I’m building a remote transport-based set of types. I want to expose human-friendly text labels to the Puppet DSL interface, but the actual REST API wants numeric ids. I’m trying to figure out the slickest way to do that. worst case I can just query for them as needed, but if I can leverage the type/resource abstraction model for it that would be great

  • to solve this use case, this would need a function to take the label <-> id mapping from the catalog and push it into the actual API resources through a hidden attribute. This doesn't sound too hot architecturally, so maybe consider a different solution for this where a provider can predeclare a query in the type (similar to autorequire) to have some data from the catalog available in the context.

[DavidS]

Possible Implementation Path notes thanks to @hlindberg:

https://github.com/puppetlabs/puppet/blob/e36a82e8f620a3e846a97c18d3d0bb6076812fce/lib/puppet/parser/compiler.rb#L154-L201 is main sequencing of compile actions. After finish and prune_catalog, the data structure is ready to transmit. validate_final should still run after post-processing to ensure that modifications didn't invalidate invariants (might need multiple validation runs to isolate impact of individual processors).

The @catalog at that point still contains full Puppet::Parser::Resource objects and a lot of calculated information. To reduce programming complexity for post-processors, some back-and-forth translation layer to pure data would be useful.

Related idea: transforming serialized catalog out-of-proc instead of loading up puppetserver. Advantages: line-format already exists, not bound to ruby, happens after all current processing
Disadvantages: line-format is tabulated (hard to unravel without puppet code), still need to hook into puppetserver somehow, a lot of data gets shipped back-and-forth, conceptually should be part of compiler.

https://tickets.puppetlabs.com/browse/PUP-8088 has implementation of some compiler hooks attached as example of how this could be implemented.

UX/OpX still TBD