doc/website: v0.24 release notes (#2860)

* doc/website: v0.24 release notes

* slash

* doc: add testing pages

---------

Co-authored-by: Ariel Mashraki <[email protected]>

doc/md/testing/migrate.mdx

@@ -0,0 +1,270 @@
+---
+title: Testing Migrations
+slug: /testing/migrate
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import LoginRequired from '../components/login-required.mdx'
+
+The `atlas migrate test` command allows you to write tests for your schema migrations. This feature
+enables you to test logic in your migrations in a minimal and straightforward way. The command is similar to
+[`atlas schema test`](/testing/schema) but is focused on testing schema migrations.
+
+:::info Login Required
+Migrations testing works only for logged-in users, free and paid. Use the following command to use this feature:
+
+```shell
+atlas login
+```
+
+:::
+
+### Introduction
+
+Atlas migrations testing is inspired by the popular databases in the way they test their public (and private) APIs. The structure
+is defined in HCL files (suffixed with a `.test.hcl`), but the underlying testing logic is written in plain SQL. The
+following document describes the different structure options, flags, and capabilities supported by the testing framework.
+
+#### Flags
+
+* `--dir` the URL to the migration directory, by default it is `file://migrations`.
+* `--dev-url` - a [URL](/concepts/url) to the [_Dev-Database_](../concepts/dev.mdx) to run the tests on.
+* `--run` (optional) - run only tests matching the given regexp.
+
+#### Examples
+
+<Tabs>
+<TabItem value="dir" label="Test Directories">
+
+```shell
+atlas migrate test --dev-url "docker://postgres/15/dev" .
+```
+
+</TabItem>
+<TabItem value="file" label="Test Files">
+
+```shell
+atlas migrate test --env local migrate.test.hcl
+```
+
+</TabItem>
+</Tabs>
+
+### The `test "migrate"` block
+
+The `test "migrate" "<name>"` block describes a test case. The second label defines the test case name, and the following
+arguments are supported:
+
+- `skip` (bool) - indicates whether the test case should be executed or skipped (can be computed). Defaults to `false`.
+
+Every test case starts with the zero state of the migration directory, and calls to the `migrate` command
+blocks migrate it to the specified version. At the end of the execution, Atlas cleans up the dev-database and prepares
+it for the next test case, regardless of the test result.
+
+#### Example
+
+```hcl
+test "migrate" "20240613061102" {
+  # Migrate to version 20240613061046.
+  migrate {
+    to = "20240613061046"
+  }
+  # Insert some test data.
+  exec {
+    sql = "insert into users (name) values ('Ada Lovelace')"
+  }
+  # Migrate to version 20240613061102.
+  migrate {
+      to = "20240613061102"
+  }
+  # Verify the correctness of the data migration.
+  exec {
+    sql = "select first_name,last_name from users"
+    output = "Ada, Lovelace"
+  }
+}
+```
+:::info
+* Unlike [schema test](/testing/schema) cases, migration tests cannot be run in parallel.
+* The `skip` argument or the `--run` flag can be used to test only migrations that are relevant to the current
+  development. For example, testing only the latest migration.
+:::
+
+A test case is composed of zero or more commands that are executed in order, and it is aborted if any of the commands fail.
+The supported commands are:
+
+### `migrate` command
+
+The `migrate` command expects a migration version to migrate to. The strategy for testing a specific version is to
+migrate to the version before it, insert test data, migrate to the tested version to ensure it passes, and then run
+assertions to verify its correctness.
+
+```hcl title="migrate.test.hcl"
+test "migrate" "20240613061102" {
+  # Migrate to one version before the tested version.
+  migrate {
+    to = "20240613061046"
+  }
+  # Insert some test data.
+  exec {
+    sql = "insert into users (name) values ('Ada Lovelace')"
+  }
+  # Migrate to the tested version.
+  migrate {
+      to = "20240613061102"
+  }
+  # Verify the correctness of the data migration.
+  exec {
+    sql = "select first_name,last_name from users"
+    output = "Ada, Lovelace"
+  }
+}
+```
+
+### `exec` command
+
+The `exec` command expects an SQL statement to pass. If `output` or `match` is defined, the output of the SQL statement
+is compared to the expected value.
+
+* `sql` (string) - the SQL statement to execute.
+* `format` (optional) - the format of the output (default: `csv`). Can be `table` or `csv`.
+* `output` (optional) - the expected output of the SQL statement.
+* `match` (optional) - a regular expression to match the output of the SQL statement.
+
+```hcl title="migrate.test.hcl"
+test "migrate" "20240613061046" {
+  migrate {
+    to = "20240613061046"
+  }
+  # Expected exec to pass.
+  exec {
+    sql = <<SQL
+INSERT INTO t VALUES (1, 'one');
+INSERT INTO t VALUES (2, 'two');
+SQL
+  }
+  # Expected exec to pass and output
+  # be equal to the expected table.
+  exec {
+    sql = "SELECT a, b FROM t;"
+    format = table
+    output = <<TAB
+ a |  b
+---+-----
+ 1 | one
+ 2 | two
+TAB
+  }
+}
+```
+
+### `catch` command
+
+The `catch` command expects an SQL statement to fail. If `error` is defined, the error message is compared to the expected
+value.
+
+* `sql` (string) - the SQL statement to execute.
+* `error` (optional) - matches the error message.
+
+```hcl title="migrate.test.hcl"
+test "migrate" "20240613061046" {
+  migrate {
+    to = "20240613061046"
+  }
+  catch {
+    sql   = "SELECT 1+"
+    error = "incomplete input"
+  }
+}
+```
+
+### `assert` command
+
+The `assert` command expects an SQL statement to pass and the output to be a single row (+column) with a true value.
+
+* `sql` (string) - the SQL statement to execute.
+* `error_message` (optional) - the error message to display if the assertion fails.
+
+```hcl title="migrate.test.hcl"
+test "migrate" "20240613061046" {
+  migrate {
+    to = "20240613061046"
+  }
+  assert {
+    sql = "SELECT json_valid('{')"
+    error_message = "expects a valid JSON"
+  }
+}
+```
+
+### `log` command
+
+The `log` command logs a message to the test output. It can be useful to report the progress of the test case.
+
+* `message` (string) - the message to log.
+
+```hcl title="migrate.test.hcl"
+test "migrate" "20240613061046" {
+  migrate {
+    to = "20240613061046"
+  }
+  log {
+    message = "Seeded the database"
+  }
+}
+```
+
+### Input Variables
+
+Test files can be parameterized using variables, and their values can be set through the [`atlas.hcl`](../atlas-schema/projects)
+config file. For example, given this test file:
+
+```hcl title="migrate.test.hcl"
+variable "seed_file" {
+  type = string
+}
+
+test "migrate" "seed" {
+  migrate {
+    to = "20240613061046"
+  }
+  exec {
+    sql = "SELECT seed('${var.seed_file}')"
+  }
+}
+```
+
+Test config can be defined on the `env` block (or globally) and executed using the `--env` flag:
+
+```hcl title="atlas.hcl"
+env "dev" {
+  src = "<schema to test>"
+  dev = "<docker-based dev-url>"
+
+// highlight-start
+  # Test configuration for local development.
+  test {
+    migrate {
+      src = ["migrate.test.hcl"]
+      vars = {
+        seed_file = "filename.sql"
+        variable2 = var.name
+        variable3 = data.external.value
+      }
+    }
+  }
+// highlight-end
+}
+```
+
+```
+atlas migrate test --env dev
+```
+
+:::info Input Variables
+
+[Input variables](../atlas-schema/input-variables) can be defined statically per environment, injected from the CLI using
+the `--var` flag, or computed from a [data source](../atlas-schema/projects#data-sources).
+
+:::