Skip to content

Iterations on the Developer Experience for writing specs #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

solsson
Copy link
Contributor

@solsson solsson commented May 11, 2021

Feature branch goal:

  • Make specs driven development more attractive than ad-hoc manual testing. If we succeed, backend PRs will include specs that provide knowledge transfer and can be used for automated regression testing.

@solsson
Copy link
Contributor Author

solsson commented May 11, 2021

Next experiment after #26:

  • Focus on a CI/CD environment where we want the asserted backends to keep running
  • But the specs should exit 0 after X consecutive passes of all specs
  • An initial wait of Y seconds before starting specs could be useful to reduce test noise
  • A configurable interval between reruns would also be useful to reduce noise

Unlike in dev environments where we aim for one-liners, during CI/CD it's trivial to maintain a set of environment variables for the container.

The reason exit is so important is that with the current generation of kubernetes-assert we're having issues with specs that stay around for very long and thus produce lots of test data and other types of noise/costs.

solsson added a commit that referenced this pull request May 19, 2021
@solsson
Copy link
Contributor Author

solsson commented May 20, 2021

From the evaluation internally we've observed that:

  • It's a clear improvement that specs stop running upon success
    • Specs that produce test data no longer fills our database :)
    • Logs get easier to follow
  • Actually it's also common that both dev and CI/CD results in failed specs that keep running, at least until the next working day. There is no improvement for that scenario.
  • We tricked ourselves, based on ystack example specs that specs and app should have separate skaffold yamls
    • But they should have the same so that developers:
      • Get one stream of logs to follow
      • Don't need two terminals
      • Are always encouraged to be test driven during skaffold dev

@solsson
Copy link
Contributor Author

solsson commented May 20, 2021

An initial wait of Y seconds before starting specs could be useful to reduce test noise

Relatively low priority.

A configurable interval between reruns would also be useful to reduce noise

Relatively low priority. 10 s is a very good default.

But the specs should exit 0 after X consecutive passes of all specs
... it's also common that both dev and CI/CD results in failed specs that keep running

Can we find criterias and runtime changes so that spec containers always exit?

  • If all specs have passed X times we exit 0
    • Should lead to a Completed status in kubectl get pods.
  • Evaluate a max time for the specs container, after which it exits non-zero if there are still test failures
    • Can this be cobined with restart policy so that the pod doesn't crash loop but instead we get an Error status.
    • How do we prevent exit when someone is actually developing?
      • Max time countdown should be reset on actual file watch change
      • We can define "actually developing" as editing the specs at least once within the max time interval.

We tricked ourselves, based on ystack example specs that specs and app should have separate skaffold yamls

The examples we run should dev both backend ans specs in the same skaffold dev command.

If we succeed with the above, the y-assert hack in ystack could be replaced with something that basically does kubectl get pods with some selector on specs' labels and checks the pod status.

darclander and others added 3 commits May 24, 2021 23:21
Timeout and delay should now be implemented for the cluster. Have not had enough time to see if it works properly. The logic should be there though. The image is not exited but there is a condition for where it should.
This reverts commit 66e447a.
Update jest.kubernetes-assertions-reporter.js
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants