Skip to content

Commit

Permalink
Introduce Magic Migrate
Browse files Browse the repository at this point in the history
The goal of magic migrate is to simplify versioned metadata storage.

## The problem

Cloud native build packs use toml to store metadata about a layer between builds. We commonly use this to store information like what ruby version was downloaded or a sha of some kind. On the next build we can look at this information to determine if some expensive process (like downloading a binary) can be skipped. Essentially we treat it like a cache key.

If we cannot load (deserialize) the old metadata into the currently requested structure then the default behavior is to clear the cache. This means that either the programmer must be careful to never make backwards incompatible changes to the metadata, or risk triggering a cache invalidation.

Now consider that we cannot guarantee that the cache was generated from the last version of the buildpack. Someone might deploy, then wait several years before deploying again.

The classic Ruby buildpack has this problem. It's cache is unversioned so if a mistake is made in the cache structure or contents in one version, then the fix must be hardcoded and checked on every subsequent deploy of every future version https://github.com/heroku/heroku-buildpack-ruby/blob/453b13983b638d68d9d65ab89d36a2fc18128e4a/lib/language_pack/ruby.rb#L1270-L1332.

## Introducing magic migrate

Magic migrate doesn't make these problem go away, instead it makes the problem easier to reason about. 

When the schema of the metadata changes, the programmer can introduce a new struct and tell rust how to migrate from one version to the next using either `From` or `TryFrom` (if fallible). Then they use the corresponding magic migrate trait to tell rust how to walk this chain backwards. Now when we try to load data from disk it will try to load the latest struc, if it can't it will go to the one before, and so-on. Once it finds the original serialized struct, it converts it forwards, one step at a time until we arrive at the currently desired struct.

Now instead of trying to hold all possible cache state in mind from all possible versions of the code, the programmer only needs to know how to make each conversion one by one.
  • Loading branch information
schneems committed Jan 8, 2024
1 parent c9840fd commit 6856d33
Show file tree
Hide file tree
Showing 4 changed files with 388 additions and 1 deletion.
4 changes: 4 additions & 0 deletions commons/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Changelog for commons features

## Added

- Introduce the `magic_migrate` module to aid with migrating metatadata bewteen versions https://github.com/heroku/buildpacks-ruby/pull/246

## 1.0.0

### Changed
Expand Down
2 changes: 1 addition & 1 deletion commons/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,10 @@ walkdir = "2"
ascii_table = { version = "4", features = ["color_codes"] }
const_format = "0.2"
fun_run = "0.1"
toml = "0.8"

[dev-dependencies]
indoc = "2"
filetime = "0.2"
toml = "0.8"
libcnb-test = "0.17.0"
pretty_assertions = "1"
1 change: 1 addition & 0 deletions commons/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,6 @@ pub mod display;
pub mod gem_version;
pub mod gemfile_lock;
pub mod layer;
pub mod magic_migrate;
pub mod metadata_digest;
pub mod output;
Loading

0 comments on commit 6856d33

Please sign in to comment.