PSS : Add `fs` and `inmem` state storage implementationd to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

SarahFrench · 2025-10-20T19:10:04Z

This PR enables E2E testing with pluggable state storage:

Implements PSS support in the grpcwrap package, which is needed to allow provider-like code in Core to behave like it's a real separate provider plugin.
Implements an inmem and fs state store in the v6 version of terraform-provider-simple
- Note: the filesystem PSS implementation doesn't include locking 🙈
Adds E2E tests using the two PSS implementations, showing an init followed by an apply command.

Target Release

N/A

Rollback Plan

If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

This change is user-facing and I added a changelog entry.
This change is not user-facing.

internal/builtin/providers/terraform/provider.go

SarahFrench · 2025-10-20T19:16:24Z

internal/provider-simple-v6/state_store_inmem.go

+// InMemStoreSingle allows 'storing' state in memory for the purpose of testing.
+//
+// "Single" reflects the fact that this implementation does not use any global scope vars
+// in its implementation, unlike the current inmem backend. HOWEVER, you can test whether locking
+// blocks multiple clients trying to access the same state at once by creating multiple instances
+// of backend.Backend that wrap the same provider.Interface instance.
+type InMemStoreSingle struct {
+	states stateMap
+	locks  lockMap


For context, the inmem backend uses these global vars:

terraform/internal/backend/remote-state/inmem/backend.go

Lines 24 to 30 in efb2a1c

// we keep the states and locks in package-level variables, so that they can be

// accessed from multiple instances of the backend. This better emulates

// backend instances accessing a single remote data store.

var (

states stateMap

locks lockMap

)

Like the code comment says, multiple backends can attempt to access the same state by using the same providers.Interface instance to create them.

SarahFrench · 2025-10-24T11:34:46Z

.github/workflows/checks.yml

        run: |
-          TF_ACC=1 go test -v ./internal/command/e2etest
+          TF_TEST_EXPERIMENT=1 TF_ACC=1 go test -v ./internal/command/e2etest


Something to bear in mind is that E2E tests also run in the context of the build workflow. I think it'd best to avoid enabling experiments to leak to that workflow, and instead have experimental tests skipped when running in that context.

…mmands

…nfig parsing needs to be explicitly added

… can use already parsed config

…eds to be refactored

…ly commands

…in use within operations backend

…to make relationship to (Meta).Backend more obvious.

…tations, pass through requests to matching store based on type name.

… implementation; add embeddable unimplementedProviderInterface struct

…orm provider. Stop people using it unless TF_ACC is set.

…pace state already.

…ore.

…n terraform provider

…they can be reused.

…ther work to allow PSS to be used with other commands.

…state storage.

This provider is not accessible to users so it's unnecessary.

Seeing as I was in this part of the code during this branch's work

SarahFrench added the no-changelog-needed Add this to your PR if the change does not require a changelog entry label Oct 20, 2025

SarahFrench commented Oct 20, 2025

View reviewed changes

SarahFrench changed the title ~~PSS : Add inmem state storage implementation to the builtin terraform provider~~ PSS : Add inmem state storage implementation to the builtin terraform provider, update grpcwrap package, use PSS implementation in E2E test Oct 24, 2025

SarahFrench force-pushed the pss/add-builtin-inmem-state-storage-implementation branch from a4887d9 to ef6dc39 Compare October 24, 2025 11:04

SarahFrench commented Oct 24, 2025

View reviewed changes

SarahFrench force-pushed the pss/add-builtin-inmem-state-storage-implementation branch 2 times, most recently from 0f01ff6 to 2fadb1e Compare October 24, 2025 11:38

SarahFrench added 22 commits November 3, 2025 11:05

Add a generic method for loading an operations backend in non-init co…

c5f536e

…mmands

Refactor commands to use new prepareBackend method: group 1

1313635

Refactor commands to use new prepareBackend method: group 2, where co…

4601771

…nfig parsing needs to be explicitly added

Refactor commands to use new prepareBackend method: group 3, where we…

a0eb4f0

… can use already parsed config

Additional, more nested, places where logic for accessing backends ne…

ad8edf3

…eds to be refactored

Remove duplicated comment

c91188d

Add test coverage of (m *Meta) prepareBackend()

2d7f197

Add TODO related to using plans for backend/state_store config in app…

b115aae

…ly commands

Add testStateStoreMockWithChunkNegotiation test helper

bccd461

Add assertions to tests about the backend (remote-state, local, etc) …

145f30b

…in use within operations backend

Stop prepareBackend taking locks as argument

3ca1085

Code comment in prepareBackend

8fb4e87

Replace c.Meta.prepareBackend with c.prepareBackend

3ba93c6

Change c.Meta.loadSingleModule to c.loadSingleModule

a35ecc0

Rename (Meta).prepareBackend to (Meta).backend, update godoc comment …

3378510

…to make relationship to (Meta).Backend more obvious.

Revert change from config.Module to config.Root.Module

ed75a51

Allow the builtin terraform provider to contain multiple PSS implemen…

1618814

…tations, pass through requests to matching store based on type name.

Make it easier to use provider.Interface as interface for state store…

1dd57a5

… implementation; add embeddable unimplementedProviderInterface struct

Add initial implementation of inmem state store to the builtin terraf…

5d5fea5

…orm provider. Stop people using it unless TF_ACC is set.

Add method for getting an inmem store instance that has default works…

b2e5bf9

…pace state already.

Copy some tests from the inmem backend and use for the inmem state st…

d4458b7

…ore.

Simplify code to avoid map of state stores, for now

15f775b

SarahFrench added 16 commits November 3, 2025 11:05

WIP simple

1186dd1

Implement inmem state store in provider-simple-v6, remove from builti…

2bdb391

…n terraform provider

Move PSS chunking-related constants into the pluggable package, so …

493ce15

…they can be reused.

Implement PSS-related methods in grpcwrap package

4f834fe

Update e2e test - works as expected but is blocked at apply step by o…

1bb4291

…ther work to allow PSS to be used with other commands.

Fix issues in test fixture, rename it

9b6cf2b

Update test - remove steps that are impossible to perform with inmem …

cfce283

…state storage.

Stop gating the inMem state store behind TF_ACC in the simple6 provider

c28daa3

This provider is not accessible to users so it's unnecessary.

Skip E2E test for PSS unless experiments enabled

d742402

Enable experiments in E2E tests in automation

8c87e95

Add missing space to error message text

125204e

Fix TF_TEST_EXPERIMENT => TF_TEST_EXPERIMENTS

e1998ca

Ensure state stores are configured with a suggested chunk size from Core

9a3db61

Small changes to inmem E2E test

0c5275f

Add filesystem state store (no locking) and E2E test

f26f779

Replace ioutil.ReadDir with os.ReadDir

8f77b19

Seeing as I was in this part of the code during this branch's work

SarahFrench force-pushed the pss/add-builtin-inmem-state-storage-implementation branch from 727ae19 to 8f77b19 Compare November 3, 2025 11:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PSS : Add `fs` and `inmem` state storage implementationd to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

PSS : Add `fs` and `inmem` state storage implementationd to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

SarahFrench commented Oct 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

SarahFrench Oct 20, 2025

Uh oh!

SarahFrench Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	// we keep the states and locks in package-level variables, so that they can be
	// accessed from multiple instances of the backend. This better emulates
	// backend instances accessing a single remote data store.
	var (
	states stateMap
	locks lockMap
	)

PSS : Add fs and inmem state storage implementationd to the builtin simplev6 provider, update grpcwrap package, use PSS implementation in E2E test #37790

Are you sure you want to change the base?

PSS : Add fs and inmem state storage implementationd to the builtin simplev6 provider, update grpcwrap package, use PSS implementation in E2E test #37790

Conversation

SarahFrench commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Target Release

Rollback Plan

Changes to Security Controls

CHANGELOG entry

Uh oh!

Uh oh!

SarahFrench Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

SarahFrench Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

PSS : Add `fs` and `inmem` state storage implementationd to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

PSS : Add `fs` and `inmem` state storage implementationd to the builtin `simplev6` provider, update `grpcwrap` package, use PSS implementation in E2E test #37790

SarahFrench commented Oct 20, 2025 •

edited

Loading