Pro 6471 migrations #401

BoDonkey · 2024-09-18T19:12:18Z

Summary

Summarize the changes briefly, including which issue/ticket this resolves. If it closes an existing Github issue, include "Closes #[issue number]"
The PR adds a section on writing migrations to the docs. Closes PRO-6471.

What are the specific steps to test this change?

For example:

Run the website and log in as an admin

Open a piece manager modal and select several pieces

Click the "Archive" button on the top left of the manager and confirm that it should proceed

Check that all pieces have been archived properly

What kind of change does this PR introduce?

(Check at least one)

Make sure the PR fulfills these requirements:

It includes a) the existing issue ID being resolved, b) a convincing reason for adding this feature, or c) a clear description of the bug it resolves
The changelog is updated
Related documentation has been updated
Related tests have been updated

If adding a new feature without an already open issue, it's best to open a feature request issue first and wait for approval before working on it.

Other information:

linear · 2024-09-18T19:12:20Z

PRO-6471 Create a guide to writing migrations for common cases

boutell · 2024-09-18T20:02:46Z

docs/guide/writing-migrations.md

+
+While the migration function can be added as an anonymous function as the second argument to the `add()`method, they can also be defined in the `methods(self)` customization function of the module. This can provide for a cleaner `init(self)` function, but is a matter of preference.
+
+Example adding the migration to `init(self)`


How about a trailing colon?

Overlooked and added!

boutell · 2024-09-18T20:05:27Z

docs/guide/writing-migrations.md

+    return {
+      async alignImages(self) {
+        return self.apos.migration.eachWidget({
+          type: 'image'


This is the criteria object used to find the documents, it has nothing to do with the widget type. If you want to find this widget no matter where it lies across all document types, use an empty criteria object, which is the norm. If this widget only ever occurs in one document type you can speed things up a bit by leveraging that here in the criteria object.

Got it. I thought I saw an example where this seemed to be the widget name, but now I can't find it. Makes more sense the way you explain it. Changed.

docs/guide/writing-migrations.md

boutell · 2024-09-18T20:06:38Z

docs/guide/writing-migrations.md

+
+## Adding a Missing Property to Existing Widgets
+
+Similar to updating pieces and pages, you can use the `eachWidget` helper to add or remove properties from widgets of a specific type. This is useful when updating the schema of a widget across all pages or pieces.


Worth noting that eachWidget will work no matter how the widget has been nested, e.g. in object fields, array fields and areas within other widgets.

boutell · 2024-09-18T20:07:55Z

docs/guide/writing-migrations.md

+};
+```
+
+- The `eachWidget` method iterates over **every** widget in **every** area in **every** document. For this reason, you should carefully ensure that the criteria for passing the widget to the iterator is specific to only the widgets that need to be altered.


We should water this last sentence way down because you usually can't and have to accept that a widget migration takes some time to scan all documents. You can speed it up only if you know only certain page or doc types ever contain that widget in their schema, and in that case you can do a query on type, but you must stress that it is the document type and not the widget type.

boutell · 2024-09-18T20:08:52Z

docs/guide/writing-migrations.md

+- In our `criteria` argument we pass a type of `image` to return all instances of `image` widgets that are found.
+- In the iterator we check if the `alignment` property exists, and if not, we use `$set` to add it.
+
+The `iterator` in an `eachWidget` method gets three arguments. In addition to the document, `doc`, where the widget is found, it also receives the `widget` object that will be modified and the `dotPath`. The `dotPath` argument represents the location of the current widget within the document's structure, using a "dot notation" format. It allows you to trace exactly where the widget is nested within its parent area, such as `main.content[0]`, where `main` is the area, `content` is the widget array, and `[0]` is the first widget in that array. This simplifies the process of pointing the MongoDB operation at the correct widget within a document.


It'll look like main.content.0 actually, but as long as you honor it and use it when calling $set it doesn't matter.

boutell · 2024-09-18T20:09:03Z

docs/guide/writing-migrations.md

+    return {
+      async migrate(self) {
+        return self.apos.migration.eachWidget({
+          type: 'video'


boutell · 2024-09-18T20:09:33Z

docs/guide/writing-migrations.md

+- We use `$unset` to remove the `border` property from the widget.
+
+## Additional Migrations
+While the examples above use `eachDoc` and `eachWidget` to iterate over and modify documents, you're welcome to use any MongoDB APIs you're familiar with to perform migrations. For instance, if your migration needs are simple and easily expressed through MongoDB's query capabilities, methods like `updateMany` can be more efficient than iterating over every document individually.


It is worth mentioning that the first two examples above (the eachDoc ones) can also be written with updateMany, but that this is not always practical.

boutell · 2024-09-20T12:48:22Z

docs/guide/writing-migrations.md

@@ -0,0 +1,262 @@
+# Writing Migrations
+
+Migrations in ApostropheCMS allow you to make targeted changes to your database, ensuring that your data stays in sync with the evolving structure of your code. They are particularly useful when new code deployments introduce changes to the underlying data model or schema, such as adding new fields, removing deprecated properties, or correcting inconsistencies in existing content. In this guide, we will walk through how to write migrations to add or remove properties from existing pieces and widgets.


In this coming release we are shipping logic to automatically add a default value for any new schema field that has a def property or is of a type that has its own def, which covers most types (for instance, for string the fallback def is ''). So the need to write migrations just for a brand new field alone is about to drop off considerably.

Doubling down on this because my work to automatically add default values has merged to main and release day is Wednesday, so it makes sense to address it here:

If your goal is to ensure a newly-added field in the schema will be present in the database for existing documents with its default value (as specified by def, or the fallback def of the field type such as the empty string for string fields), then you do not need to write a migration. As of version 4.x, this is automatic. However if you need to transform the existing content of the database in another way, such as renaming or removing a property or transforming a number to a string, et cetera, then migrations are the right tool for you.

boutell · 2024-09-20T12:49:54Z

docs/guide/writing-migrations.md

+
+## Removing a Property from Existing Documents
+
+If you need to remove a property, you can use `$unset`. Here’s an example that removes a `temporaryNote` property from all `default` page types:


I agree we should cover how to unset, but please add an info note stating that it is often wiser to leave the data you think you "don't need anymore" in the database and simply remove it from the schema, at least at first, to make absolutely sure you won't regret deleting information.

boutell · 2024-09-30T17:26:27Z

docs/guide/writing-migrations.md

@@ -0,0 +1,262 @@
+# Writing Migrations
+
+Migrations in ApostropheCMS allow you to make targeted changes to your database, ensuring that your data stays in sync with the evolving structure of your code. They are particularly useful when new code deployments introduce changes to the underlying data model or schema, such as adding new fields, removing deprecated properties, or correcting inconsistencies in existing content. In this guide, we will walk through how to write migrations to add or remove properties from existing pieces and widgets.


Doubling down on this because my work to automatically add default values has merged to main and release day is Wednesday, so it makes sense to address it here:

If your goal is to ensure a newly-added field in the schema will be present in the database for existing documents with its default value (as specified by def, or the fallback def of the field type such as the empty string for string fields), then you do not need to write a migration. As of version 4.x, this is automatic. However if you need to transform the existing content of the database in another way, such as renaming or removing a property or transforming a number to a string, et cetera, then migrations are the right tool for you.

boutell · 2024-09-30T17:27:18Z

docs/guide/writing-migrations.md

+
+## Adding migrations
+
+In ApostropheCMS, migrations are added using the [`add(name, fn)` method](/reference/modules/migration.md#add-name-fn) of the `@apostrophecms/migration` module. One common place to add these are within the `init(self)` initialization function of your module. Each migration requires a unique name and is only run **once**. ApostropheCMS tracks which migrations have already been executed, ensuring they won’t run again across restarts or deployments.


"one common place to add these is within" right? (English is weird)

Yup. corrected.

boutell · 2024-09-30T17:30:27Z

docs/guide/writing-migrations.md

+      return self.apos.migration.eachDoc({
+        type: 'article'
+      }, async (doc) => {
+        if (doc.featured === undefined) {


I think a simple example like this is still good, but I would document it as a way to clean up if a string field was first added with no def and then you decide, as a one-time thing, to change the empty string to a more useful default later. Apostrophe can't do that automatically because it can't divine your intent and the empty string can be a legitimate value.

boutell · 2024-09-30T17:35:32Z

docs/guide/writing-migrations.md

+To apply pending migrations in a production environment, run:
+
+```bash
+node app @apostrophecms/migration:run


The task name is @apostrophecms/migration:migrate

Although migrations currently do run automatically in both development and production, it is best practice to run the @apostrophecms/migration:migrate task in production before launching the newest version of the application to serve requests. At a future time, an option to disable Apostrophe's check for needed migrations on ordinary invocations in production may be offered as an optimization.

(Yeaaaaa I just found out this is true two weeks ago sorry LOL)

Number 1 - changed. I guess I was rushing.
Number 2 - changed.

boutell · 2024-09-30T17:36:08Z

docs/guide/writing-migrations.md

+
+## Adding or Modifying a Property in Existing Documents
+
+When a new property needs to be added to all instances of a document type, you can use the [`eachDoc`](/reference/modules/migration.md#async-eachdoc-criteria-limit-iterator) helper provided by the migration module. This method efficiently queries documents in your collection and allows you to update them with only the necessary changes. The `eachDoc` helper takes three parameters.


When a property of all instances of a document type needs to be changed, transformed or added in a way more complicated than setting def at the time it is first added to the code...

boutell · 2024-09-30T17:39:16Z

docs/guide/writing-migrations.md

+              await self.apos.doc.db.updateOne({
+                _id: doc._id
+              }, {
+                $unset: { `${dotPath.border}`: '' }


Incorrect logic here, .border is supposed to be part of the string, not evaluated as a property of dotPath (you need to close your } sooner)

BoDonkey · 2024-09-30T19:51:10Z

I'm wondering with the new intro if we need an example of changing a string schema field to a number?

boutell

All good, just providing you with a real version number for the line about when def beomes automatic

boutell · 2024-10-01T14:00:18Z

docs/guide/writing-migrations.md

@@ -1,10 +1,10 @@
 # Writing Migrations

-Migrations in ApostropheCMS allow you to make targeted changes to your database, ensuring that your data stays in sync with the evolving structure of your code. They are particularly useful when new code deployments introduce changes to the underlying data model or schema, such as adding new fields, removing deprecated properties, or correcting inconsistencies in existing content. In this guide, we will walk through how to write migrations to add or remove properties from existing pieces and widgets.
+Migrations in ApostropheCMS allow you to make targeted changes to your database, ensuring that your data stays in sync with the evolving structure of your code. If your goal is to ensure a newly-added field in the schema will be present in the database for existing documents with its default value (as specified by def, or the fallback def of the field type such as the empty string for string fields), then you do not need to write a migration. As of version 4.x, this is automatic. However, if you need to transform the existing content of the database in another way, such as renaming or removing a property or transforming a number to a string, then migrations are the right tool for you. In this guide, we will walk through how to write migrations to add or remove properties from existing pieces and widgets.


Shipping tomorrow, so you can write 4.8.0 here

boutell · 2024-10-01T14:02:21Z

docs/guide/writing-migrations.md

            await self.apos.doc.db.updateOne({
              _id: doc._id
            }, {
-              $set: { description: 'No description available' }
+              $set: { copyright: '©2024 ApostropheCMS. All rights reserved.' }


Still not a frequent use case anymore (because def will automatically get propagated), but I do think we need a simple example and it's valid, so let's go with it.

boutell · 2024-10-01T14:15:33Z

docs/guide/writing-migrations.md

@@ -249,7 +244,7 @@ module.exports = {
              await self.apos.doc.db.updateOne({
                _id: doc._id
              }, {
-                $unset: { `${dotPath.border}`: '' }
+                $unset: { `${dotPath}.border`: '' }


... Huh. I see that your empty string "value" is consistent with how mongodb examples handle this, I guess they like a natural "falsy"/"empty" value for whatever the type was, even though the value does not matter at all with $unset. 👍

BoDonkey added 2 commits September 17, 2024 16:12

Add initial migration guide

b4fd555

Update and finish migration examples

f0a6318

BoDonkey requested a review from boutell September 18, 2024 19:12

boutell requested changes Sep 20, 2024

View reviewed changes

Add responses to first comments

888c75e

BoDonkey requested a review from boutell September 30, 2024 13:36

boutell requested changes Sep 30, 2024

View reviewed changes

Add responses to second comments.

0aad4ae

BoDonkey requested a review from boutell September 30, 2024 19:51

boutell requested changes Oct 1, 2024

View reviewed changes

Add responses to third comments

ae68bd3

BoDonkey requested a review from boutell October 2, 2024 12:16

boutell approved these changes Oct 2, 2024

View reviewed changes

BoDonkey merged commit f9e9a91 into main Oct 2, 2024

BoDonkey deleted the pro-6471-migrations branch October 2, 2024 13:16


		While the migration function can be added as an anonymous function as the second argument to the `add()`method, they can also be defined in the `methods(self)` customization function of the module. This can provide for a cleaner `init(self)` function, but is a matter of preference.

		Example adding the migration to `init(self)`


		## Adding a Missing Property to Existing Widgets

		Similar to updating pieces and pages, you can use the `eachWidget` helper to add or remove properties from widgets of a specific type. This is useful when updating the schema of a widget across all pages or pieces.

		@@ -0,0 +1,262 @@
		# Writing Migrations

		Migrations in ApostropheCMS allow you to make targeted changes to your database, ensuring that your data stays in sync with the evolving structure of your code. They are particularly useful when new code deployments introduce changes to the underlying data model or schema, such as adding new fields, removing deprecated properties, or correcting inconsistencies in existing content. In this guide, we will walk through how to write migrations to add or remove properties from existing pieces and widgets.


		## Removing a Property from Existing Documents

		If you need to remove a property, you can use `$unset`. Here’s an example that removes a `temporaryNote` property from all `default` page types:


		## Adding migrations

		In ApostropheCMS, migrations are added using the [`add(name, fn)` method](/reference/modules/migration.md#add-name-fn) of the `@apostrophecms/migration` module. One common place to add these are within the `init(self)` initialization function of your module. Each migration requires a unique name and is only run once. ApostropheCMS tracks which migrations have already been executed, ensuring they won’t run again across restarts or deployments.


		## Adding or Modifying a Property in Existing Documents

		When a new property needs to be added to all instances of a document type, you can use the [`eachDoc`](/reference/modules/migration.md#async-eachdoc-criteria-limit-iterator) helper provided by the migration module. This method efficiently queries documents in your collection and allows you to update them with only the necessary changes. The `eachDoc` helper takes three parameters.

Pro 6471 migrations #401

Pro 6471 migrations #401

Conversation

BoDonkey commented Sep 18, 2024

Summary

What are the specific steps to test this change?

What kind of change does this PR introduce?

Make sure the PR fulfills these requirements:

linear bot commented Sep 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoDonkey commented Sep 30, 2024

boutell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment