Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: include test dependency and random test information #64

Merged
merged 12 commits into from
Sep 12, 2024
89 changes: 89 additions & 0 deletions src/content/docs/testing/testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -418,3 +418,92 @@ abstract class TestTag {
```

:::

## Do not share state between tests

Tests should not share state between them to ensure they remain independent, reliable, and predictable.

When tests share state—such as variables, test's outcome and order can influence another's, leading to inconsistent results and making it difficult to identify the root cause of failures.
alestiago marked this conversation as resolved.
Show resolved Hide resolved

<Tabs>
<TabItem label="Good ✅">

```dart
class _Counter {
int value = 0;
void increment() => value++;
void decrement() => value--;
}

void main() {
group(_Counter, () {
late _Counter counter;

setUp(() => counter = _Counter());

test('increment', () {
counter.increment();
expect(counter.value, 1);
});

test('decrement', () {
counter.decrement();
expect(counter.value, -1);
});
});
}
```

</TabItem>
<TabItem label="Bad ❗️">

```dart
class _Counter {
int value = 0;
void increment() => value++;
void decrement() => value--;
}

void main() {
group(_Counter, () {
final _Counter counter = _Counter();

test('increment', () {
counter.increment();
expect(counter.value, 1);
});

test('decrement', () {
counter.decrement();
// The expectation only succeeds when the previous test executes first.
expect(counter.value, 0);
});
});
}
```

</TabItem>
</Tabs>

## Use random test ordering

Running tests in an arbitrary (random) order is a crucial practice to identify and eliminate flaky tests, specially during continuous integration.

Flaky tests are those that pass or fail inconsistently without changes to the codebase, often due to unintended dependencies between tests.

By running tests in random order, these hidden dependencies are more likely to be exposed, as any reliance on the order of test execution becomes clear when tests fail unexpectedly.

This practice ensures that tests do not share state or rely on the side effects of previous tests, leading to a more robust and reliable test suite. Overall, the tests become easier to trust and reduce debugging time caused by intermittent test failures.

<Tabs>
<TabItem label="Good ✅">

```sh
# Randomize test ordering using the --test-randomize-ordering-seed option
flutter test --test-randomize-ordering-seed random
dart test --test-randomize-ordering-seed random
very_good test --test-randomize-ordering-seed random
```

</TabItem>
</Tabs>