Skip to content

Commit

Permalink
Update Berkeley Update Section
Browse files Browse the repository at this point in the history
  • Loading branch information
kaozenn committed May 16, 2024
1 parent 5c79a04 commit e1d1283
Show file tree
Hide file tree
Showing 22 changed files with 1,086 additions and 112 deletions.
82 changes: 82 additions & 0 deletions docs/berkeley-upgrade/appendix.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
title: Appendix
sidebar_label: Appendix
hide_title: true
description: Berkeley Upgrade Appendix
keywords:
- Berkeley
- upgrade
- appendix
---

# Appendix

## Migration from o1labs/client-sdk to mina-signer

The signing library o1labs/client-sdk was deprecated some time ago and will stop working after the upgrade. All users should upgrade with the library mina-signer https://www.npmjs.com/package/mina-signer.

Below you will find an example of how to use the mina-signer library. Please keep in mind the following:

1. Make sure to adjust the nonce to the correct nonce on the account you want to use
2. Use fee 1 MINA to get the TX submitted quickly
3. Update the “url” variable with an existing Mina Node GraphQL

```javascript
import { Client } from 'mina-signer';

// create testnet client and define hard-coded keypair

const client = new Client({ network: 'testnet' });

let privateKey = Your private key;
let publicKey = the public key - perhaps derived from the private key using -> client.derivePublicKey(privateKey);

// define and sign payment

let payment = {
from: publicKey,
to: 'to public key',
amount: 100,
nonce: 1,
fee: 1000000,
};

const signedPayment = client.signPayment(payment, privateKey);

// send payment to graphql endpoint

let url = 'https://qanet.minaprotocol.network/graphql';

let query = `mutation {
sendPayment(
input: ${objectToGraphqlQuery(signedPayment.data)},
signature: ${objectToGraphqlQuery(signedPayment.signature)}
) {
payment { id }
}
}`;

console.log('=======================');
console.log(query);
console.log('=======================');

let response = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ operationName: null, query, variables: {} }),
});
if (response.status == 200) {
let json = await response.json();
console.dir(json, { depth: null });
} else {
let text = await response.text();
console.log('Error:\n', text);
}

function objectToGraphqlQuery(obj: any) {
let json = JSON.stringify(obj, null, 2);
// removes the quotes on JSON keys
return json.replace(/\"(\S+)\"\s*:/gm, '$1:');
}

```
137 changes: 137 additions & 0 deletions docs/berkeley-upgrade/archive-migration/appendix.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
---
title: Appendix
sidebar_label: Appendix
hide_title: true
description: archive node schema changes between Mainnet and Berkeley
keywords:
- Berkeley
- upgrade
- archive migration
- appendix
- mina archive node
- archive node
---

# Appendix

## Archive node schema changes

If you are using the Archive Node database directly for your system integrations, then you should understand all the changes that might impact your applications. The most important change is that the `balances` table in the Berkeley schema will no longer exist. In the new schema, it is replaced with the table `accounts_accessed` - from an application semantics point of view, the data in `accounts_accessed` is still the same.

In the Berkeley protocol, accounts can now have the same public key but a different token_id. This means accounts are identified by both their public key and token_id, not just the public key. Consequently, the foreign key for the account in all tables is account_identifier_id instead of public_key_id.

### Schema differences
- **Removed Types**
- The options `create_token`, `create_account`, and `mint_tokens` have been removed from the user_command_type enumeration.
- Indexes Dropped
- We've removed several indexes from tables, this may affect how you search and organize data:
- `idx_public_keys_id`
- `idx_public_keys_value`
- `idx_snarked_ledger_hashes_value`
- `idx_blocks_id`
- `idx_blocks_state_hash`
- **Table Removed**
- The `balances` table is no longer available.
- **New Tables Added**
- We've introduced the following new tables:
- `tokens`
- `token_symbols`
- `account_identifiers`
- `voting_for`
- `protocol_versions`
- `accounts_accessed`
- `accounts_created`
- `zkapp_commands`
- `blocks_zkapp_commands`
- `zkapp_field`
- `zkapp_field_array`
- `zkapp_states_nullable`
- `zkapp_states`
- `zkapp_action_states`
- `zkapp_events`
- `zkapp_verification_key_hashes`
- `zkapp_verification_keys`
- `zkapp_permissions`
- `zkapp_timing_info`
- `zkapp_uris`
- `zkapp_updates`
- `zkapp_balance_bounds`
- `zkapp_nonce_bounds`
- `zkapp_account_precondition`
- `zkapp_accounts`
- `zkapp_token_id_bounds`
- `zkapp_length_bounds`
- `zkapp_amount_bounds`
- `zkapp_global_slot_bounds`
- `zkapp_epoch_ledger`
- `zkapp_epoch_data`
- `zkapp_network_precondition`
- `zkapp_fee_payer_body`
- `zkapp_account_update_body`
- `zkapp_account_update`
- `zkapp_account_update_failures`
- **Updated Tables**
- The following tables have been updated
- `timing_info`
- `user_commands`
- `internal_commands`
- `epoch_data`
- `blocks`
- `blocks_user_commands`
- `blocks_internal_commands`

### Differences per table
- **`timing_info`**
- Removed columns:
- `token`
- `initial_balance`
- **`user_commands`**
- Removed columns:
- `fee_token`
- `token`
- **`internal_commands`**
- Removed columns:
- `token`
- Renamed column
- `command_type` to `type`
- **`epoch_data`**
- Added columns:
- `total_currency`
- `start_checkpoint`
- `lock_checkpoint`
- `epoch_length`
- **`blocks`**
- Added columns:
- `last_vrf_output`
- `min_window_density`
- `sub_window_densities`
- `total_currency`
- `global_slot_since_hard_fork`
- `global_slot_since_genesis`
- `protocol_version_id`
- `proposed_protocol_version_id`
- Removed column:
- `global_slot`
- **`blocks_user_commands`**
- Removed columns:
- `fee_payer_account_creation_fee_paid`
- `receiver_account_creation_fee_paid`
- `created_token`
- `fee_payer_balance`
- `source_balance`
- `receiver_balance`
- Added index:
- `idx_blocks_user_commands_sequence_no`
- **`blocks_internal_commands`**
- Removed columns:
- `receiver_account_creation_fee_paid`
- `receiver_balance`
- Added indexes:
- `idx_blocks_internal_commands_sequence_no`
- `idx_blocks_internal_commands_secondary_sequence_no`

### Rosetta API new operations

The Berkeley upgrade introduces two new operation types:
- `zkapp_fee_payer_dec`
- `zkapp_balance_change`
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ keywords:
- Berkeley
- upgrade
- archive migration
- installing
- installing
- prerequisites
- mina archive node
- archive node
---

The archive node Berkeley migration package is sufficient for satisfying the migration from Devnet/Mainnet to Berkeley.
However, it has some limitations. For example, the migration package does not migrate a non-canonical chain and it skips orphaned blocks that are not part of a canonical chain.
However, it has some limitations. For example, the migration package does not migrate a non-canonical chain and it skips orphaned blocks that are not part of a canonical chain.

To mitigate these limitations, the archive node maintenance package is available for use by archive node operators who want to maintain a copy of their Devnet and Mainnet databases for historical reasons.

Expand All @@ -35,23 +35,23 @@ We strongly encourage you to perform the migration on your own data to preserve
1. Download the Devnet/Mainnet archive data using cURL or gsutil:

- cURL:

For Devnet:
```sh
curl https://storage.googleapis.com/mina-archive-dumps/devnet-archive-dump-{date}_0000.sql.tar.gz
```
For Mainnet:

For Mainnet:
```sh
curl https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-{date}_0000.sql.tar.gz
```

To filter the dumps by date, replace `{date}` using the required `yyyy-dd-mm` format. For example, for March 15, 2024, use `2024-03-15`.

:warning: The majority of backups have the `0000` suffix. If a download with that name suffix is not available, try incrementing it. For example, `0001`, `0002`, and so on.

- gsutil:

```sh
gsutil cp gs://mina-archive-dumps/mainnet-archive-dump-2024-01-15* .
```
Expand All @@ -65,13 +65,13 @@ We strongly encourage you to perform the migration on your own data to preserve
3. Import the Devnet/Mainnet archive dump into the Berkeley database.

Run this command at the database server:

```sh
psql -U {user} -f {network}-archive-dump-{date}_0000.sql
```

The database in the dump **archive_balances_migrated** is created with the Devnet/Mainnet archive schema.

Note: This database does not have any Berkeley changes.

## Ensure the location of Google Cloud bucket with the Devnet/Mainnet precomputed blocks
Expand All @@ -84,17 +84,17 @@ The recommended method is to perform migration on your own data to preserve the

## Validate the Devnet/Mainnet database

The correct Devnet/Mainnet database state is crucial for a successful migration.
The correct Devnet/Mainnet database state is crucial for a successful migration.

[Missing blocks](/berkeley-upgrade/mainnet-database-maintenance#missing-blocks) is one the most frequent issues when dealing with the Devnet/Mainnet archive. Although this step is optional, it is strongly recommended that you verify the archive condition before you start the migration process.
[Missing blocks](/berkeley-upgrade/archive-migration/mainnet-database-maintenance#missing-blocks) is one the most frequent issues when dealing with the Devnet/Mainnet archive. Although this step is optional, it is strongly recommended that you verify the archive condition before you start the migration process.

To learn how to maintain archive data, see [Devnet/Mainnet database maintenance](/berkeley-upgrade/mainnet-database-maintenance).
To learn how to maintain archive data, see [Devnet/Mainnet database maintenance](/berkeley-upgrade/archive-migration/mainnet-database-maintenance).

## Download the migration applications

Migration applications are distributed as part of the archive migration Docker and Debian packages.

Choose the packages that are appropriate for your environment.
Choose the packages that are appropriate for your environment.

### Debian packages

Expand All @@ -118,7 +118,7 @@ To get the Docker image:
docker pull gcr.io/o1labs-192920/mina-archive-migration:3.0.1-e848ecb-{codename}
```
Where supported codenames are:
Where supported codenames are:
- bullseye
- focal
- buster
Expand All @@ -132,9 +132,9 @@ The Mina Devnet/Mainnet genesis ledger is stored in GitHub in the `mina` reposit
You can get the Berkeley schema files from different locations:
- GitHub repository from the `berkeley` branch.
- GitHub repository from the `berkeley` branch.
Note: The `berkeley` branch can contain new updates regarding schema files, so always get the latest schema files instead of using an already downloaded schema.
Note: The `berkeley` branch can contain new updates regarding schema files, so always get the latest schema files instead of using an already downloaded schema.
- Archive/Rosetta Docker from `berkeley` version
Expand All @@ -148,4 +148,4 @@ You can get the Berkeley schema files from different locations:

## Next steps

Congratulations on completing the essential preparation and verification steps. You are now ready to perform the migration steps in [Migrating Devnet/Mainnet Archive to Berkeley Archive](/berkeley-upgrade/migrating-archive-database-to-berkeley).
Congratulations on completing the essential preparation and verification steps. You are now ready to perform the migration steps in [Migrating Devnet/Mainnet Archive to Berkeley Archive](/berkeley-upgrade/archive-migration/migrating-archive-database-to-berkeley).
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Worked example of Devnet Archive Migration
sidebar_label: Worked example (devnet 2024-03-22)
title: Example of Devnet Archive Migration (Debian)
sidebar_label: Debian example (Devnet)
hide_title: true
description: A copy-paste example of how to do a Devnet migration.
keywords:
Expand All @@ -11,9 +11,11 @@ keywords:
- archive node
---

You can follow these steps that can be copy-pasted directly into a fresh Debian 11.
# Debian example

This example uses an altered two-step version of the [full simplified workflow](/berkeley-upgrade/migrating-archive-database-to-berkeley#simplified-approach).
You can follow these steps that can be copy-pasted directly into a fresh Debian 11.

This example uses an altered two-step version of the [full simplified workflow](/berkeley-upgrade/archive-migration/migrating-archive-database-to-berkeley#simplified-approach).

```sh
apt update && apt install lsb-release sudo postgresql curl wget gpg # debian:11 is surprisingly light
Expand Down Expand Up @@ -54,7 +56,6 @@ mina-berkeley-migration-script initial \
--blocks-batch-size 100 --blocks-bucket mina_network_block_data \
--network devnet


# now, do a final migration

gsutil cp gs://mina-archive-dumps/devnet-archive-dump-2024-03-22_2050.sql.tar.gz .
Expand Down
Loading

0 comments on commit e1d1283

Please sign in to comment.