Create and upload a deb repository to staging automatically #5240

faern · 2023-10-06T12:59:25Z

Add to buildserver-build.sh so that it upon finishing building a release (stable or beta) do the following:

Construct a deb repository locally at deb/$app_version where the just built artifacts are added to
Upload the constructed repository to staging repository server and replacing the existing repository there at /deb/beta and if it's a stable release also to /deb/stable.

The prepared repo is kept locally. So once this has been tested to work the person doing the release can publish it to production with the help of publish-linux-repositories.sh.

This change is

linear · 2023-10-06T12:59:28Z

DES-360 Update artifact upload script to upload to apt update servers

With hand holding from infra (Victor), update the buildserver-upload.sh script to prepare and upload appropriate files to releases.mullvad.net in such a way that install and automatic updates from them work on apt based distros.

The steps to prepare and upload a release for APT supposedly goes something like this according to Victor:

dpkg-sig --sign builder mullvad-vpn.deb

reprepro -V --basedir /var/www/cdn/repository/ includedeb jammy /path/to/the/.deb_file
reprepro -V --basedir /var/www/cdn/repository/ includedeb focal /path/to/the/.deb_file
reprepro -V --basedir /var/www/cdn/repository/ includedeb bullseye /path/to/the/.deb_file

rsync -azP /var/www/cdn/repository/ buildreleases.devmole.eu:/var/www/cdn/repository/

Debian-howto from Victor:

### ON BUILDING MACHINE - APP TEAM ###
# dependencies
sudo apt install reprepro dpkg-sig
# create same folder structure as on the server for easy rsync
mkdir deb
mkdir deb/conf
#find gpg KEY-ID and export pubkey to file
gpg -k
gpg --armor --export $KEY-ID | sudo tee deb/mullvad-keyring.asc
# Add distributions file that covers all supported distros - needs to be kept up2date
vi deb/conf/distributions

###################### Add to file: ########################
Origin: cdn.devmole.eu
Label: apt repository
Codename: jammy
Architectures: amd64
Components: main
Description: Devmole package repository for Debian/Ubuntu
SignWith: 1AB64AED77EB5969 # This is your signature obviously - get it from gpg --list-sigs user-id
Pull: jammy

Origin: cdn.devmole.eu
Label: apt repository
Codename: focal
Architectures: amd64
Components: main
Description: Devmole package repository for Debian/Ubuntu
SignWith: 1AB64AED77EB5969
Pull: focal

Origin: cdn.devmole.eu
Label: apt repository
Codename: bullseye
Architectures: amd64
Components: main
Description: Devmole package repository for Debian/Ubuntu
SignWith: 1AB64AED77EB5969
Pull: bullseye

#################### END OF FILE ###########################

# Run
# export GPG_TTY=$(tty) <--- probably not needed but included just in case
dpkg-sig --sign builder $DEBFILE
reprepro -V --basedir deb/ includedeb jammy /path/to/the/$DEBFILE
reprepro -V --basedir deb/ includedeb focal /path/to/the/$DEBFILE
reprepro -V --basedir deb/ includedeb bullseye /path/to/the/$DEBFILE
# Remove package in case we need to
reprepro -V --basedir deb/ remove jammy firefox

##### Testing on CLIENT

sudo curl -fsSLo /usr/share/keyrings/mullvad-keyring.asc https://cdn.devmole.eu/repository/deb/mullvad-keyring.asc

echo "deb [signed-by=/usr/share/keyrings/mullvad-keyring.asc arch=$( dpkg --print-architecture )] https://cdn.devmole.eu/repository/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/mullvad.list

apt update && apt install mullvad-vpn

## FUTURE, I'll let you know
#sudo curl -fsSLo /usr/share/keyrings/mullvad-keyring.asc https://repository.devmole.eu/deb/mullvad-keyring.asc
#echo "deb [signed-by=/usr/share/keyrings/mullvad-keyring.gpg arch=$( dpkg --print-architecture )] https://cdn.devmole.eu/repository/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/mullvad.list

#Then rsync the deb-folder :)

raksooo

Reviewed 4 of 4 files at r2.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @faern)

ci/buildserver-config.sh line 6 at r2 (raw file):

# the scripts where they are used.

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

This will override the SCRIPT_DIR in buildserver-build.sh since this is running in the same scope. They are running from the same location so it doesn't really make a difference though. Removing this line would make this script use the SCRIPT_DIR from the script sourcing this one which is a bit cleaner IMO but it would still cause trouble if they were located in different directories.

ci/buildserver-config.sh line 9 at r2 (raw file):

# Which gpg key to sign things with
export CODE_SIGNING_KEY_FINGERPRINT="A1198702FC3E0A09A9AE5B75D5A1D4F266DE8DDF"

Is there a benefit of exporting these variables? Since this file is sourced it runs under the same scope and they would be available when not exported as well.

faern

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @raksooo)

ci/buildserver-config.sh line 9 at r2 (raw file):

Previously, raksooo (Oskar Nyberg) wrote…

Is there a benefit of exporting these variables? Since this file is sourced it runs under the same scope and they would be available when not exported as well.

We do in env.sh. I'm 100% sure I tried this and it did not export when sourcing and not using export. Dammit. I do not want. :bash: 🔨

faern

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @raksooo)

ci/buildserver-config.sh line 6 at r2 (raw file):

Previously, raksooo (Oskar Nyberg) wrote…

This will override the SCRIPT_DIR in buildserver-build.sh since this is running in the same scope. They are running from the same location so it doesn't really make a difference though. Removing this line would make this script use the SCRIPT_DIR from the script sourcing this one which is a bit cleaner IMO but it would still cause trouble if they were located in different directories.

I do not think this script should rely on someone else before it to set SCRIPT_DIR. So how do I compute some intermediate variables I need here without exporting them?

faern

Reviewable status: 3 of 4 files reviewed, 2 unresolved discussions (waiting on @raksooo)

ci/buildserver-config.sh line 9 at r2 (raw file):

Previously, faern (Linus Färnstrand) wrote…

We do in env.sh. I'm 100% sure I tried this and it did not export when sourcing and not using export. Dammit. I do not want. :bash: 🔨

I do think we need to export them. Otherwise the values would not propagate to subshells. Something I think we might need even if I don't have a case in front of me right now.

raksooo

Reviewable status: 3 of 4 files reviewed, 1 unresolved discussion

ci/buildserver-config.sh line 9 at r2 (raw file):

Previously, faern (Linus Färnstrand) wrote…

I do think we need to export them. Otherwise the values would not propagate to subshells. Something I think we might need even if I don't have a case in front of me right now.

You're right!

raksooo

Reviewable status: 3 of 4 files reviewed, 1 unresolved discussion (waiting on @faern)

ci/buildserver-config.sh line 6 at r2 (raw file):

Previously, faern (Linus Färnstrand) wrote…

I do not think this script should rely on someone else before it to set SCRIPT_DIR. So how do I compute some intermediate variables I need here without exporting them?

Maybe we should just use SCRIPT_DIR and let it override the parents SCRIPT_DIR. As you said that's what we do in env.sh and maybe it's not worth the time now to figure this out.

raksooo

Reviewed 4 of 4 files at r4.
Reviewable status: complete! all files reviewed, all discussions resolved

raksooo

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @faern)

-- commits line 4 at r5:
Should builds be buildserver or something?

faern

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @raksooo)

-- commits line 4 at r5:

Previously, raksooo (Oskar Nyberg) wrote…

Should builds be buildserver or something?

It's a bit strangely formulated maybe, but it's not a typo. It means "build an apt repository when performing app builds and store locally on the build server.

"on builds" means "when performing app builds" in this case.

Suggestion for improvement? Also, just commit message, so not very important

ci/buildserver-config.sh line 6 at r2 (raw file):

Previously, raksooo (Oskar Nyberg) wrote…

Maybe we should just use SCRIPT_DIR and let it override the parents SCRIPT_DIR. As you said that's what we do in env.sh and maybe it's not worth the time now to figure this out.

Fixed

raksooo

Reviewed all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved

-- commits line 4 at r5:

Previously, faern (Linus Färnstrand) wrote…

It's a bit strangely formulated maybe, but it's not a typo. It means "build an apt repository when performing app builds and store locally on the build server.

"on builds" means "when performing app builds" in this case.

Suggestion for improvement? Also, just commit message, so not very important

Ah okay, my brain had a difficult time parsing that. But not that important as you said 👍

Build an apt/deb repository on builds and store locally on the build server. Also push it to development infra instantly. Push to staging if it's a release. And store locally for manual push to production later

faern requested a review from raksooo October 6, 2023 12:59

raksooo reviewed Oct 10, 2023

View reviewed changes

faern commented Oct 11, 2023

View reviewed changes

faern force-pushed the update-artifact-upload-script-to-upload-to-apt-update-des-360 branch from c0cc078 to 0936bbf Compare October 11, 2023 13:59

faern commented Oct 11, 2023

View reviewed changes

raksooo reviewed Oct 11, 2023

View reviewed changes

raksooo reviewed Oct 12, 2023

View reviewed changes

faern force-pushed the update-artifact-upload-script-to-upload-to-apt-update-des-360 branch from 0936bbf to 39c8076 Compare October 12, 2023 08:17

raksooo approved these changes Oct 12, 2023

View reviewed changes

faern force-pushed the update-artifact-upload-script-to-upload-to-apt-update-des-360 branch 2 times, most recently from 0a7b665 to 63188f5 Compare October 13, 2023 12:47

faern marked this pull request as ready for review October 13, 2023 13:50

raksooo reviewed Oct 17, 2023

View reviewed changes

faern commented Oct 17, 2023

View reviewed changes

raksooo approved these changes Oct 18, 2023

View reviewed changes

Add APT repository preparation and upload scripts

32d182b

Build an apt/deb repository on builds and store locally on the build server. Also push it to development infra instantly. Push to staging if it's a release. And store locally for manual push to production later

faern force-pushed the update-artifact-upload-script-to-upload-to-apt-update-des-360 branch from 63188f5 to 32d182b Compare October 20, 2023 14:18

faern merged commit b2a9781 into main Oct 20, 2023
7 checks passed

faern deleted the update-artifact-upload-script-to-upload-to-apt-update-des-360 branch October 20, 2023 14:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create and upload a deb repository to staging automatically #5240

Create and upload a deb repository to staging automatically #5240

faern commented Oct 6, 2023 •

edited

Loading

linear bot commented Oct 6, 2023

raksooo left a comment

faern left a comment

faern left a comment

faern left a comment

raksooo left a comment

raksooo left a comment

raksooo left a comment

raksooo left a comment

faern left a comment

raksooo left a comment

Create and upload a deb repository to staging automatically #5240

Create and upload a deb repository to staging automatically #5240

Conversation

faern commented Oct 6, 2023 • edited Loading

linear bot commented Oct 6, 2023

raksooo left a comment

Choose a reason for hiding this comment

faern left a comment

Choose a reason for hiding this comment

faern left a comment

Choose a reason for hiding this comment

faern left a comment

Choose a reason for hiding this comment

raksooo left a comment

Choose a reason for hiding this comment

raksooo left a comment

Choose a reason for hiding this comment

raksooo left a comment

Choose a reason for hiding this comment

raksooo left a comment

Choose a reason for hiding this comment

faern left a comment

Choose a reason for hiding this comment

raksooo left a comment

Choose a reason for hiding this comment

faern commented Oct 6, 2023 •

edited

Loading