Go test runner

[lox]

Description

Implement a go test runner

Changes

Adds a go test runner, with test listing, splitting, etc.

Testing

Pretty extensive go tests.

[niceking]

Hey @lox! Thanks for this 😁 We'll give it a go (🥁 ) locally and send back any thoughts!

[lox]

I ran into all sorts of problems getting it to actually run and lack of viz into the current pipelines, etc. Lemme know if there is interest over there for a chat, I'm happy to keep contributing on the go side.

[wooly]

Hi @lox

Thanks again for submitting this PR!

We had a small mobbing session today to see if we could get it running with the tests in this repository but ran into a few problems. We tried running the client against a local server (with the Go support change outlined below), but were unable to get things working. The command we used was go run main.go with the relevant environment variables configured.

I ran into all sorts of problems getting it to actually run and lack of viz into the current pipelines

We've added support server-side for Go runners, so you should be able to run this on your prod pipelines shortly to verify that everything is working.

Would you be able to give this a try and report back if things are working?

If you want a hand getting set up then we can schedule a call.

Cheers!

[lox]

We've added support server-side for Go runners, so you should be able to run this on your prod pipelines shortly to verify that everything is working.

Thanks, can you help me understand what this means?

[wooly]

Thanks, can you help me understand what this means?

If you have a test suite collecting Go test results (using gotestsum or some other means to send JUnit data to Test Engine), you can run bktec locally or in a pipeline (you'll have to build your own bktec binary for this) to run an end-to-end test of Go support in bktec. If you want to run things in a pipeline, then this guide should help you get started.

To run this locally, you could export the following env vars after substituting <some-org-slug>, <some-access-token>, and <some-suite-slug> with your own values from Buildkite production:

BUILDKITE_BUILD_ID=1234
BUILDKITE_STEP_ID=5678
BUILDKITE_ORGANIZATION_SLUG=<some-org-slug>
BUILDKITE_PARALLEL_JOB=0
BUILDKITE_PARALLEL_JOB_COUNT=2
BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN=<some-access-token>
BUILDKITE_TEST_ENGINE_DEBUG_ENABLED=true
BUILDKITE_TEST_ENGINE_RESULT_PATH=results.json
BUILDKITE_TEST_ENGINE_SUITE_SLUG=<some-suite-slug>
BUILDKITE_TEST_ENGINE_TEST_RUNNER=go
BUILDKITE_TEST_ENGINE_RETRY_COUNT=0

NB You'll need to change the BUILDKITE_STEP_ID each time you run go run main.go to ensure unique plans are generated with each run.

[niceking]

Hey @lox just checking in about this!

Were the extra instructions helpful? We could also make a time to set up a zoom or something :) or we could try and get it working more on our end if you don't wanna keep pushing this along!

[lox]

I'll give it a shot this week!

[nprizal]

Hi @lox

I’ve looked into this and discovered some complexities/limitations in splitting Go tests by test file. Currently, the Test Engine lacks the capability to extract test file information from a Go test result. This means that our server can’t create an optimal (evenly partitioned) splitting by test file, unlike other test frameworks. We’re considering splitting Go tests by test or package, which needs a different implementation compared to other test frameworks. To do that, we first need to change our server side implementation to enable us to create an optimal partition in a scalable way. We'll let you know when we have more information.

[lox]

It's quite unclear to me what aspects are server-side and what are in this client, is there docs somewhere on what that boundry is?