fix: keys migration (document deserialization error)

[mikeplotean]

Description

Introduces keys migration to convert jwk keys documents to new format.

Fixes #625

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• I have performed a self-review of my code
• If it is a core feature, I have added thorough tests
• Do we need to implement analytics?
• Will this be part of a product update? If yes, please write one phrase about this update.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[notion-workspace]

Database key document format

[waltkb]

Shouldn't it maybe only be executed on demand?

[mikeplotean]

Shouldn't it maybe only be executed on demand?

it's executed only on data matching the criteria (in this case, jwk keyType having the jwk field as a string). what do you mean by demand?

[waltkb]

WalletKeys.selectAll(), each of which it will do some JSON operation on to check if it's JWK document or JWK object

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[waltkb]

Shouldn't it maybe only be executed on demand?

Migration system:

• chronological ordered (by monotonic counter) list of migrating functions, where each of the the functions has a sequential ID (e.g. migration 1, 2, 3, 4, 5, 6, 7)
• a table that holds the latest run migration id (e.g. latest migration 4)
• a system that on startup runs the migrations with IDs higher than the latest run migration (e.g. 5, 6, 7), in order

e.g.: (explicit types for reference, most not needed)

1. migration functions (of course names and other stuff can also be given in more advanced version)

val migrations = listOf<() -> Unit>(
    { /* migration function code, or reference thereto */ }, // migration 1
    { /* migration function code, or reference thereto */ }, // migration 2
    { /* migration function code, or reference thereto */ }, // migration 3
    { /* migration function code, or reference thereto */ }, // migration 4
)

2. Table: standard exposed table

object Migrations : Table("migrations") {
    val latestRunMigration = integer("latest_run_migration")
    override val primaryKey: PrimaryKey = PrimaryKey(latestRunMigration)
}

3. simple system

val totalMigrations = migrations.size // list from above

// this function is called on startup, after the database is connected, but before the webserver handles connections (in `init = {` at ServiceInitialization)
fun runNeededMigrations() {
    val lastRunMigration = getLatestRunMigration() // select on Migration table
    if (totalMigrations > lastRunMigration) {
        // additional migrations have to run
        migrations.drop(latestMigration).forEachIndexed { index, migration -> // only run migrations that have not yet been run
            // next migration
            migration.invoke()
            setLastestRunMigration(lastRunMigration + index + 1)
        }
    }
}

[waltkb]

see above: Only run migrations when necessary

[mikeplotean]

see above: Only run migrations when necessary

should we implement an in-house data migration tool?
cc @philpotisk