Conflict between plugin framework 1.4 and sdk 2.28 with tfprotov6.ProviderServer

[mschuchard]

Module version

1.4.1

Note this issue also occurs in 1.4.0.

Relevant provider source code

var TestAccProtoV6ProviderFactories = map[string]func() (tfprotov6.ProviderServer, error){
    "providerName": providerserver.NewProtocol6WithError(NewStruct("test")),
}

Terraform Configuration Files

n/a

Debug Output

This is equivalent to logging at default level.

Expected Behavior

This is really difficult actually to describe beyond "it should not error". I would suggest one of the options below:

1. Upgrading the plugin framework module to 1.4.x does not break acceptance testing without accompanying documentation about backwards incompatibilities/upgrade guide. The CHANGELOG notes contain a reference to a PR at Add GetMetadata RPC to provider protocol terraform#33739. This mysteriously claims:

This is a followup to the new provider server capability to make the GetProviderSchema RPC optional.

which conflicts with the error message stating it must be defined.

2. The acceptance testing uses the specified version of the tfproto as the error message states v5 instead of the correct v6. Alternatively the error message contains the incorrect information about the version (and possibly also the mandatory versus optional nature of GetMetadata; see 1.)

3. The Terraform plugin framework does not utilize the SDK for this functionality (probably more of a "stretch goal" here).

4. The plugin framework is updated in some other way to resolve the conflict with sdk v2.28.

Actual Behavior

# github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema
../../go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/provider.go:511:9: cannot use NewGRPCProviderServer(p) (value of type *GRPCProviderServer) as tfprotov5.ProviderServer value in return statement: *GRPCProviderServer does not implement tfprotov5.ProviderServer (missing method GetMetadata)

Steps to Reproduce

TF_ACC=1 go test -v /path/to/acceptnace_tests

References

already referenced inline

[bflad]

Hi @mschuchard 👋 Thank you for reporting this and apologies the release notes around this were not clearer. We'll keep that in mind in the future.

Upfront, this issue should be resolvable by updating your terraform-plugin-sdk/v2 dependency to v2.29.0:

go get github.com/hashicorp/terraform-plugin-sdk/[email protected]
go mod tidy

At this point, the terraform-plugin-sdk/v2 Go module is relatively frozen at this point except to pick up compatibility issues like this particular case. More about the why below.

So what happened here? We introduced an additional operation (GetMetadata) in the Terraform Plugin Protocol. This change first happens in terraform-plugin-go, the lowest level Go module for provider development. That Go module is not versioned the same as the higher level SDK Go modules (terraform-plugin-sdk and terraform-plugin-framework), but both depend on it, so introducing any interface changes means that both SDKs need to be updated at the same time. Consequently that lockstep dependency upgrade requirement is passed onto provider codebases themselves. Unfortunately there's no way for us in the terraform-plugin-framework Go module to also denote the required terraform-plugin-sdk minimum version, since there is no actual direct or indirect dependency between the two Go modules.

If your provider is no longer developed with terraform-plugin-sdk based resources, but that dependency is still being used for your acceptance testing, we would recommend migrating to the terraform-plugin-testing, which should be relatively straightforward to remove the direct dependency. Even if you are not building resources with terraform-plugin-sdk anymore though, there's still a confusing lingering dependency when using terraform-plugin-testing today. When we split the acceptance testing functionality of terraform-plugin-sdk into the separate terraform-plugin-testing, we wanted to ensure that migration was as easy as possible, which meant keeping its few references to terraform-plugin-sdk types in certain places. In the future though, there will be a major version upgrade of terraform-plugin-testing to fully remove the lingering terraform-plugin-sdk dependency: hashicorp/terraform-plugin-testing#186

In the meantime, sorry for the hassle in this in-between period. If you're using a dependency management system like Dependabot, there's functionality that can suggest bundled updates to help prevent this in the future: https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#groups

Just to walk through your specific expectations in your original issue:

Upgrading the plugin framework module to 1.4.x does not break acceptance testing without accompanying documentation about backwards incompatibilities/upgrade guide.

Yes, until we resolve hashicorp/terraform-plugin-testing#186. we will try to include this sort of additional compatibility information in release notes.

The acceptance testing uses the specified version of the tfproto as the error message states v5 instead of the correct v6. Alternatively the error message contains the incorrect information about the version (and possibly also the mandatory versus optional nature of GetMetadata; see 1.)

../../go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/provider.go:511:9: cannot use NewGRPCProviderServer(p) (value of type *GRPCProviderServer) as tfprotov5.ProviderServer value in return statement: *GRPCProviderServer does not implement tfprotov5.ProviderServer (missing method GetMetadata)

This error is controlled by the Go compiler and is actually correct, it was just unfortunately confusing for the situation. terraform-plugin-sdk/[email protected] does not satisfy the interface implementation required by the terraform-plugin-go upgrade caused by upgrading terraform-plugin-framework in this case.

The Terraform plugin framework does not utilize the SDK for this functionality (probably more of a "stretch goal" here).

terraform-plugin-framework has no dependency on terraform-plugin-sdk itself. You can verify that by looking at the go.mod file for the Go module here: https://github.com/hashicorp/terraform-plugin-framework/blob/v1.4.1/go.mod

Acceptance testing for your provider codebase will either directly depend on terraform-plugin-sdk or indirectly depend on terraform-plugin-sdk via terraform-plugin-testing, which we plan to resolve in the future as mentioned above.

The plugin framework is updated in some other way to resolve the conflict with sdk v2.28.

This can be achieved by remaining on the prior version of terraform-plugin-framework. Go modules cannot specify dependency requirements for those not in their dependency graph, as far as I'm aware.

If there is some particular reason why upgrading terraform-plugin-sdk/[email protected] to v2.29.0 is an issue, please reach out.

[mschuchard]

Thank you for the resolution as updating the indirect dependency on the sdk to 2.29 in the go.mod indeed resolved the issue. I was a bit perturbed initially that it required updating an indirect dependency for best practices reasons, but your explanation of the uncontrollable "dependency hell" situation here absolves your responsibility for sure.

If your provider is no longer developed with terraform-plugin-sdk based resources, but that dependency is still being used for your acceptance testing, we would recommend migrating to the terraform-plugin-testing, which should be relatively straightforward to remove the direct dependency.

Except all of this is with the terraform-plugin-testing module. As you may recall the v1.5.1 release arose from an issue I reported related to a customer's plugin I developed where I could not specify provider block inputs with env. I have no idea what causes the indirect dependency on the sdk, and I had assumed (incorrectly) it was related to terraform-plugin-go. I see now according to your explanation why it is an indirect dependency, and now I look forward to this v2.0 release.

Glad this was such a quick recommendation for you and thanks for the additional information. This issue also probably should have been raised in terraform-plugin-testing instead. If you can/want to migrate it to that repo then that would be totally cool too.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.