Use non-computed property value comparison for matching detailed diff set inputs to plan

[VenelinMartinov]

This PR reworks the way we compute the detailed diff for set type elements in order to handle sets with computed properties better.

Context

There are two reasons it is challenging to produce the detailed diff for set type properties:

• The planning process re-orders the elements of a set.
• The engine does not have access to the post-plan properties but can only display diff previews in terms of OldState and NewInputs.

These two reasons means that once we compute the detailed diff in terms of OldState to PlannedState, we need to do a bit of work to transform it into a diff in terms of OldState and NewInputs. Matching PlannedState values to NewInputs values is especially challenging for elements which contain Computed properties as these can change during the planning process. The previous implementation used the hashes of set elements to do the matching but that completely failed for set elements with Computed properties, as the Computed property will change the hash.

New implementation

This implementation instead uses a technique borrowed from opentofu, which is used for a similar purpose. We now do a fuzzy match on the PlannedState value to the NewInputs value, which allows for differences in Computed properties. Non-Computed properties still need to match exactly.

Example

For the schema

"foo": Set{
  Elem: {
     "bar": {Type: string, Optional: true}
     "baz": {Type: string, Computed: true}
  }
}

and the inputs
[{bar: val1}, {bar: val2}]
changing to
[{bar: val1}, {bar: val3}]
we might observe the plan becoming
[{bar: val3, baz: comp3}, {bar: val1, baz: comp1}]

We now need to correctly match the elements to their corresponding inputs, in order to correctly communicate the diff to the engine in terms of OldState and NewInput:

{bar: val3, baz: comp3} should match {bar: val3} (the second element in the input)
and {bar: val1, baz: comp1} should match {bar: val1}

Testing

I've added additional integration tests around Computed in #2740 as well as unit tests here. I'll annotate any left-over issues in the recorded tests to be resolved.

Changes up to 4ba30d9 have the code changes and 4ba30d9 has the test recordings.

Release

Note that this is feature flagged behind Accurate Previews

fixes #2652
fixes #2528

[VenelinMartinov]

This change is part of the following stack:

• Use non-computed property value comparison for matching detailed diff set inputs to plan #2761 ◀

_{Change managed by git-spice.}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.69%. Comparing base (f666912) to head (d36172e).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2761      +/-   ##
==========================================
- Coverage   68.72%   68.69%   -0.03%     
==========================================
  Files         322      322              
  Lines       41340    41422      +82     
==========================================
+ Hits        28411    28455      +44     
- Misses      11333    11365      +32     
- Partials     1596     1602       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[t0yv0]

Left some last comments but this is generally LGTM!

[pulumi-bot]

This PR has been shipped in release v3.99.0.