From 86c3ffb38672758b90adc89235fdd7da3e47cf93 Mon Sep 17 00:00:00 2001
From: Arthur Gousset <46296830+arthurgousset@users.noreply.github.com>
Date: Fri, 24 May 2024 20:12:26 +0100
Subject: [PATCH] Reduce build time when running foundry migrations (#11003)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* chore(protocol/migrations_sol): adds time tracking

To measure how long it takes to run the entire script from start to finish.

* chore(protocol/migrations_sol): cuts migration time by 35%

Worked as expected, but stopped working because of what I think is a cache issue.

Latest best time to run script (with improvement): 338 sec or	5.6	min
Improvement margin: -34% or	2.9	min

Committing this WIP to do a clean checkout and see if it's a cache issue.

* chore(protocol/contracts): adds `@celo-contracts/` import

This uses remappings instead of relative import `./`

* fix(protocol): updates `@celo-contracts` remapping

* refactor(protocol/migrations_sol): adds `deploy_libraries.sh` script

Currently running `create_and_migrate_anvil_devchain.sh` successfully builds the libraries (fast) as expected.

It then fails with:

```sh
Deploying libraries...
Deploying library: AddressSortedLinkedListWithMedian
Error:
Found incompatible Solidity versions:
test-sol/governance/validators/IntegerSortedLinkedListMock-8.sol (>=0.8.0 <0.8.20) imports:
    contracts-0.8/common/linkedlists/IntegerSortedLinkedList.sol (>=0.8.0 <0.8.20)
    lib/openzeppelin-contracts8/contracts/utils/math/SafeMath.sol (^0.8.0)
    contracts/common/linkedlists/SortedLinkedList.sol (^0.5.13)
    lib/openzeppelin-contracts/contracts/math/SafeMath.sol (^0.5.0)
    contracts/common/linkedlists/LinkedList.sol (^0.5.13)
    lib/openzeppelin-contracts/contracts/math/SafeMath.sol (^0.5.0)
```

* fix(protocol/contracts-0.8): updates `@celo-contracts-8`

Previously I mistakenly updated to `@celo-contracts` which is on the wrong solidity version.

* fix(protocol/migrations_sol): changes directory to deploy contracts

Previously the script tried to deploy contracts in the `protocol/` but the build output in the temp directory.

* fix(protocol/contracts): undo `@celo-` imports

Using `@celo-contracts` and `@celo-contracts-8` syntax with remappings broke the truffle build process.

Truffle only allows imports (with or without remapping) that are resolved within `node_modules`.

That means I don't think I can't use a directory that is not in `node_modules` (with or without remapping).

> Truffle supports dependencies installed via NPM. To import contracts from a dependency, use the following syntax
> `import "somepackage/SomeContract.sol";`
> Here, `somepackage` represents a package installed via NPM, and `SomeContract.sol` represents a Solidity source file provided by that package.

Source: https://archive.trufflesuite.com/docs/truffle/how-to/compile-contracts/#importing-dependencies-via-file-name

> Since the path didn't start with `./`, Truffle knows to look in your project's `node_modules` directory for the `example-truffle-library` folder. From there, it resolves the path to provide you the contract you requested.

Source: https://archive.trufflesuite.com/docs/truffle/how-to/package-management-via-npm/#within-your-contracts

I decided to revert the changes to the import and the need for remappings, and instead try to make the library compilation work with relative paths.

At this point
1. the truffle compilation works with

```sh
# within protocol directory
yarn build:sol
```

2. the Foundry migrations work

```sh
# within protocol directory
./migrations_sol/create_and_migrate_anvil_devchain.sh
```

* chore(protocol/migrations_sol): moves `mkdir` command up

The script was breaking on CI:

```sh
mkdir: cannot create directory ‘/home/runner/work/celo-monorepo/celo-monorepo/packages/protocol/.tmp/libraries’: No such file or directory
```

Source: https://github.com/celo-org/celo-monorepo/actions/runs/9223759766/job/25377654339?pr=11003#step:18:78

That's expected since the temp directory is only created after the "deploy libraries" script is called.

* chore(protocol/migrations_sol): removes `TODO` comment

* chore(protocol/migrations_sol): removes unused library commands

I tested this, and these extra commands are not used or needed.
Also removes `TODO` comment.
---
 packages/protocol/foundry.toml                |  2 +-
 .../create_and_migrate_anvil_devchain.sh      | 71 ++++++---------
 .../migrations_sol/deploy_libraries.sh        | 87 +++++++++++++++++++
 3 files changed, 115 insertions(+), 45 deletions(-)
 create mode 100644 packages/protocol/migrations_sol/deploy_libraries.sh

diff --git a/packages/protocol/foundry.toml b/packages/protocol/foundry.toml
index 95fe05401e1..f309779040c 100644
--- a/packages/protocol/foundry.toml
+++ b/packages/protocol/foundry.toml
@@ -14,7 +14,7 @@ remappings = [
     'forge-std-8/=lib/celo-foundry-8/lib/forge-std/src/',
     '@celo-contracts-8=contracts-0.8/',
     '@openzeppelin/contracts8/=lib/openzeppelin-contracts8/contracts/',
-    '@celo-contracts=contracts/'
+    '@celo-contracts/=contracts/'
 ]
 
 no_match_contract = "RandomTest"
diff --git a/packages/protocol/migrations_sol/create_and_migrate_anvil_devchain.sh b/packages/protocol/migrations_sol/create_and_migrate_anvil_devchain.sh
index fbbda837206..dc49fc4af1f 100755
--- a/packages/protocol/migrations_sol/create_and_migrate_anvil_devchain.sh
+++ b/packages/protocol/migrations_sol/create_and_migrate_anvil_devchain.sh
@@ -1,20 +1,34 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
-# Compile everything
-forge build
+# Keeping track of start time to measure how long it takes to run the script entirely
+START_TIME=$SECONDS
 
 export ANVIL_PORT=8546
 
 # TODO make this configurable
 FROM_ACCOUNT_NO_ZERO="f39Fd6e51aad88F6F4ce6aB8827279cffFb92266" # This is Anvil's default account (1)
-FROM_ACCOUNT="0x$FROM_ACCOUNT_NO_ZERO"
+export FROM_ACCOUNT="0x$FROM_ACCOUNT_NO_ZERO"
+
+# Create temporary directory
 TEMP_FOLDER="$PWD/.tmp"
+mkdir -p $TEMP_FOLDER
 
+# Start a local anvil instance
 source $PWD/migrations_sol/start_anvil.sh
 
-source $PWD/migrations_sol/deploy_precompiles.sh
+# Deploy libraries to the anvil instance
+source $PWD/migrations_sol/deploy_libraries.sh
+echo "Library flags are: $LIBRARY_FLAGS"
+
+# Build all contracts with deployed libraries
+# Including contracts that depend on libraries. This step replaces the library placeholder
+# in the bytecode with the address of the actually deployed library.
+echo "Compiling with libraries... "
+time forge build $LIBRARY_FLAGS
 
+# Deploy precompile contracts
+source $PWD/migrations_sol/deploy_precompiles.sh
 
 echo "Setting Registry Proxy"
 REGISTRY_ADDRESS="0x000000000000000000000000000000000000ce10"
@@ -24,50 +38,19 @@ REGISTRY_OWNER_ADDRESS=$FROM_ACCOUNT_NO_ZERO
 
 echo "Setting Registry owner"
 # Sets the storage of the registry so that it has an owner we control
-# pasition is bytes32(uint256(keccak256("eip1967.proxy.admin")) - 1);
+# position is bytes32(uint256(keccak256("eip1967.proxy.admin")) - 1);
 cast rpc anvil_setStorageAt --rpc-url http://127.0.0.1:$ANVIL_PORT $REGISTRY_ADDRESS 0xb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d6103 "0x000000000000000000000000$REGISTRY_OWNER_ADDRESS"
 
-
-echo "Deploying libraries"
-LIBRARIES_PATH=("contracts/common/linkedlists/AddressSortedLinkedListWithMedian.sol:AddressSortedLinkedListWithMedian"
-                "contracts/common/Signatures.sol:Signatures"
-                "contracts/common/linkedlists/AddressLinkedList.sol:AddressLinkedList"
-                "contracts/common/linkedlists/AddressSortedLinkedList.sol:AddressSortedLinkedList"
-                "contracts/common/linkedlists/IntegerSortedLinkedList.sol:IntegerSortedLinkedList"
-                "contracts/governance/Proposals.sol:Proposals"
-)
-
-LIBRARIES=""
-
-for library in "${LIBRARIES_PATH[@]}"; do
-    library_name="${library#*:}" 
-    echo "Deploying library: $library_name"
-    create_library_out=`forge create $library --from $FROM_ACCOUNT --rpc-url http://127.0.0.1:$ANVIL_PORT --unlocked --json`
-    library_address=`echo $create_library_out | jq -r '.deployedTo'`
-    
-    LIBRARIES="$LIBRARIES --libraries $library:$library_address"
-done
-
-echo "Library flags are: $LIBRARIES"
-echo "Backing up libraries"
-
-mkdir -p $TEMP_FOLDER
-
-LIBRARIES_FILE="$TEMP_FOLDER/libraries.tx"
-rm -f $LIBRARIES_FILE
-touch $LIBRARIES_FILE
-
-echo "$LIBRARIES" > $LIBRARIES_FILE
-
+# run migrations
+echo "Running migration script... "
 # helpers to disable broadcast and simulation
 # TODO move to configuration
 BROADCAST="--broadcast"
-SKIP_SUMULATION=""
-# SKIP_SUMULATION="--skip-simulation" 
+SKIP_SIMULATION=""
+# SKIP_SIMULATION="--skip-simulation" 
 # BROADCAST=""
+time forge script migrations_sol/Migration.s.sol --tc Migration --rpc-url http://127.0.0.1:$ANVIL_PORT -vvv $BROADCAST --non-interactive --sender $FROM_ACCOUNT --unlocked $LIBRARY_FLAGS || echo "Migration script failed"
 
-echo "Compiling with libraries... "
-time forge build $LIBRARIES
-
-# run migrations
-time forge script migrations_sol/Migration.s.sol --tc Migration --rpc-url http://127.0.0.1:$ANVIL_PORT -vvv $BROADCAST --non-interactive --sender $FROM_ACCOUNT --unlocked $LIBRARIES || echo "Migration script failed"
\ No newline at end of file
+# Keeping track of the finish time to measure how long it takes to run the script entirely
+ELAPSED_TIME=$(($SECONDS - $START_TIME))
+echo "Total elapsed time: $ELAPSED_TIME seconds"
diff --git a/packages/protocol/migrations_sol/deploy_libraries.sh b/packages/protocol/migrations_sol/deploy_libraries.sh
new file mode 100644
index 00000000000..4a9c5d5ac82
--- /dev/null
+++ b/packages/protocol/migrations_sol/deploy_libraries.sh
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Name of temporary directory
+TEMP_DIR_NAME=".tmp/libraries"
+TEMP_DIR="$PWD/$TEMP_DIR_NAME"
+
+# Create a temporary directory or remove it first it if exists
+if [ -d "$TEMP_DIR" ]; then
+    echo "Removing existing temporary folder..."
+    rm -rf $TEMP_DIR
+fi
+mkdir $TEMP_DIR
+
+# Copy libraries to the directory
+LIBRARIES_PATH=("contracts/common/linkedlists/AddressSortedLinkedListWithMedian.sol:AddressSortedLinkedListWithMedian"
+                "contracts/common/Signatures.sol:Signatures"
+                "contracts/common/linkedlists/AddressLinkedList.sol:AddressLinkedList"
+                "contracts/common/linkedlists/AddressSortedLinkedList.sol:AddressSortedLinkedList"
+                "contracts/common/linkedlists/IntegerSortedLinkedList.sol:IntegerSortedLinkedList"
+                "contracts/governance/Proposals.sol:Proposals"
+)
+
+for LIB_PATH in "${LIBRARIES_PATH[@]}"; do
+    IFS=":" read -r SOURCE DEST <<< "$LIB_PATH"
+    DEST_FILE="$TEMP_DIR/$SOURCE"
+    DEST_DIR=$(dirname "$DEST_FILE")
+    mkdir -p "$DEST_DIR"
+    echo "Copying file $SOURCE to $DEST_FILE"
+    cp "$SOURCE" "$DEST_FILE"
+done
+
+# Copy dependencies of the libraries to the directory
+LIBRARY_DEPENDENCIES_PATH=(
+    "contracts/common/FixidityLib.sol"
+    "contracts/common/linkedlists/LinkedList.sol"
+    "contracts/common/linkedlists/SortedLinkedList.sol"
+    "contracts/common/linkedlists/SortedLinkedListWithMedian.sol"
+    "lib/openzeppelin-contracts/contracts/math/SafeMath.sol"
+    "lib/openzeppelin-contracts/contracts/math/Math.sol"
+    "lib/openzeppelin-contracts/contracts/cryptography/ECDSA.sol"   
+    "lib/openzeppelin-contracts/contracts/utils/Address.sol"
+    "lib/solidity-bytes-utils/contracts/BytesLib.sol"
+)
+
+# Creating two variables for better readability
+SOURCE_DIR=$PWD
+DEST_DIR=$TEMP_DIR
+
+for LIB_PATH in "${LIBRARY_DEPENDENCIES_PATH[@]}"; do
+    # Creates directory for the dependency, including any necessary parent directories
+    mkdir -p $DEST_DIR/$(dirname $LIB_PATH)
+    # Copies dependency to the newly created directory
+    cp $SOURCE_DIR/$LIB_PATH $DEST_DIR/$LIB_PATH
+done
+
+# Copy foundry config to the temporary directory
+cp $SOURCE_DIR/foundry.toml $DEST_DIR/foundry.toml
+
+# Move into the temporary directory
+pushd $TEMP_DIR
+
+# Build libraries
+echo "Building libraries..."
+forge build
+
+# Deploy libraries and building library flag
+echo "Deploying libraries..."
+export LIBRARY_FLAGS=""
+for LIB_PATH in "${LIBRARIES_PATH[@]}"; do
+    LIB_NAME="${LIB_PATH#*:}" 
+    # For example:
+    # LIB_PATH = "contracts/common/linkedlists/AddressSortedLinkedListWithMedian.sol:AddressSortedLinkedListWithMedian"
+    # LIB_NAME = AddressSortedLinkedListWithMedian
+    echo "Deploying library: $LIB_NAME"
+    create_library_out=`forge create $LIB_PATH --from $FROM_ACCOUNT --rpc-url http://127.0.0.1:$ANVIL_PORT --unlocked --json`
+    LIB_ADDRESS=`echo $create_library_out | jq -r '.deployedTo'`
+    # Constructing library flag so the remaining contracts can be built and linkeded to these libraries
+    LIBRARY_FLAGS="$LIBRARY_FLAGS --libraries $LIB_PATH:$LIB_ADDRESS"
+done
+
+# Move out of the temporary directory
+popd
+
+# Remove the temporary directory
+rm -rf $TEMP_DIR
+