Best practices for migrations

[vhpoet]

Do you have any best practices / examples documented somewhere for authorization model migrations? Say an object type or a relation has changed, one might need to migrate the tuples. There's multiple ways of doing this and I haven't found an elegant one yet. I'm not necessarily asking for a single solution, but wondering if this is something you'd be covering in the docs at some point?

[matthewpereira]

Hey @vhpoet ! This is actually a discussion a lot of us on the team have been navigating and exploring quite a bit over the last couple weeks. These are some of the scenarios we're evaluating:

• Rename type/relation
• Create new type/relation
• Remove type/relation
• Extend assignable relations of type
• Remove assignable relations of type

These sorts of conversations that involve deployment and versioning and real world usage spark ideas about the fundamental architecture and user experience of OpenFGA, so we're trying to be thoughtful with how we proceed in evaluating different options before we prescribe solutions in our docs.

For the example you mentioned, an object type or relation has changed, one pattern we're exploring is:

1. relation_a exists in your model
2. Add relation_b to your model
3. Add define relation_b as relation_a or self to your model
4. Save model and take note of version ID
5. Update your application code, and redeploy targeting the new version ID (or latest, depending on your strategy here)
6. Batch create new tuples for relation_b from old relation_a tuples
7. Delete old relation_a tuples
8. Remove relation_a from your model
9. Update relation_b definition to define relation_b as self
10. Save and redeploy as before

I'd love to hear your feedback, and to hear about any alternative patterns you might be considering as well.

[vhpoet]

Below are some of my thoughts from the past few days.

Background

Types of migrations

1. Migrating a model.
2. Migrating tuples.

Migration scenarios

1. Model changes, but tuples don't have to migrate.

• Create new type/relation.
• Extend assignable relations of type.

2. Model changes and tuples have to migrate.

• Rename type/relation.
• Remove type/relation.
• Remove assignable relations of type.

Migration strategies for Blue/Green deployment

1. Single OpenFGA instance, single store, two models (doesn't work if tuples have to be migrated).
2. Single OpenFGA instance, two stores, one model.
3. Two OpenFGA instances, each with a single store and a single model.

1	2	3

Ideal World

• Model and the tuples are migrated in a single step to avoid getting out of sync.
• For most of the use cases (renaming/removing object or relations) tuples get migrated without manually writing a migration script.
• Both migration scenarios are handled in the same way.
• Mechanism for Blue/Green deployment including a way to revert changes.

Proposed Solution

Introduce tooling for migrations as part of OpenFGA sdks (inspired by knex).

• Instead of manually creating / maintaining a model.dsl, we'd have migration files with up/down scripts which would add/remove/edit objects and their relations. The sdk would go through all migrations and generate a model.dsl that eventually gets written to the store.
• Migration files would contain exact instructions (say rename an object type from x to y) which would make automatic migrations for tuples possible. I can think of only two other ways to make tuple migrations:
  • Manually write the migrations like proposed in the above post.
  • Automatically migrate tuples based on the diff of the old and new DSL (I'm doubtful this would even work).

[jon-whit]

If we considered changing the API so that an OpenFGA Store has only a single authorization model, then it seems we could solve a lot of these problems. Using your Blue/Green Deployment scenario above, both the Blue (old) and Green (new) deployments would be referencing the same authorization model (there's only one), and if we required services to migrate tuples before renaming/removing type-definitions/relations/type-restrictions, then you could roll these out like a rolling database schema migration in lock step.

I think that model and tuple management/migrations should be treated similarly to rolling database migrations. You shouldn't take your database offline to update a column, but instead you should introduce a new column and gradually migrate to it by reading from the old column and writing to the new one and then once all replicas of your app are reading from the new column and all data has been moved (in a background job) from the old column to the new one, then you can officially cutover and drop the old column.

I think the only challenging bit might be

Mechanism for Blue/Green deployment including a way to revert changes.

A revert implies going back to a prior model (maybe one that had a relation that was recently removed). The problem in this case is that migrations involve data and schema management. If you've migrated or removed the data, then how do you recover back in time from that? Similarly, the problem is even more complex because if you added a relation and wrote some tuples and then you want to rollback your model to the last most recent one, then that would require dropping data as well (similar to dropping a column if you rollback an added column). This is why, for blue/green deployment topologies, it is recommended not to drop columns in database rollbacks but instead just omit/ignore them until it has been decided they are no longer needed. Blue/green deployments require more of a roll forward mentality instead of rollback mentality, and they don't cater well to traditional database rollback strategies, period.

This is a good article on the subject that I've read in the past and I like the way they put it.
https://www.red-gate.com/blog/how-to-avoid-rollbacks-in-database-devops

[aaguiarz]

Hi @vhpoet

Thanks for this!

I like the idea of having migration scripts. It's a complex problem to solve. For example, if the script says "rename type from x to y", we'd need to:

1. Copy tuples that have type x and use type y.
2. Whenever a tuple with type x is written, also write one with type y.
3. When we are done copying tuples, direct all traffic to the new model version.
4. Delete all tuples with type x.

Is not only a 'migration script' but also changing the behavior of the system while the migration is running (e.g. step 2).

We'd need to define a language to express different changes in the DSL (e.g. split the admin relationship to billing_admin to user_admin), and figure out the right migration strategy. Not sure if we can express any change and if we can execute any migration without downtime.

I see this as something that will be built eventually.

[vhpoet]

ok this makes a lot of sense to me, thanks for the detailed responses @matthewpereira, @aaguiarz and @jon-whit. Do you have any rough timelines for introducing tools and changes to the system to make this possible?

[aaguiarz]

Hi @vhpoet

No, we don't have a timeline yet.

[adriantam]

Hello @vhpoet , thank you for bringing the lack of documentation for migration to our attention. We have added documentations (https://openfga.dev/docs/modeling/migrating-relations) to describe the recommended steps for migrating models.

[vhpoet]

This is great, thanks @adriantam! Hope to get some tooling around this at some point too.