BATs acceptance tests?

[TJM]

While I was looking for ways to test for the storage errors, I noticed that the Hashicorp plugins tend to use "BATs" for acceptance testing. That way they are able to spin up docker containers as needed (artifactory) and use the vault binary directly instead of interfacing at the "Backend" layer in code, which feels like more like unit testing (unit tests with a real artifactory server).

I am not saying we can't keep the go-based acceptance tests, but it feels kinda strange to have direct code access when doing acceptance tests. Acceptance Tests are supposed to be "black-box" or completely separate from the code. We can write them in golang, but they need to be in a separate directory (package).

Since the "normal" user interface is the vault binary, it feels like a shell based testing system would be easier.

Thoughts?

[TJM]

our "Makefile" tasks like make setup (its individual components) are actually pretty close to acceptance tests, they just need assertions ;)

[alexhung]

Agree, more akin to the acceptance tests in Terraform provider will be ideal. i.e. the testing framework actually spawn a new test instance of the plugin, set it up with the Vault server, send the test steps (as cli text stream?), and check the results.

Alas, what HashiCorp has provided leave us doing most of the heavy lifting right now.