-
Notifications
You must be signed in to change notification settings - Fork 15
Devops Tools for Citus Projects
This repo includes all scripts and tools being used by Citus projects. Scripts are mostly developed using python. Scripts in this repository could be used both manually and in pipelines.
Installation
git clone --depth=1 https://github.com/citusdata/tools.git toolspython -m pip install -r tools/packaging_automation/requirements.txt
These are the scripts to be able to create packages for Citus Projects
This script prepares the code to be packaged to packaging process. This script should be executed before tagging to update required configurations and create required branches
python -m tools.packaging_automation.prepare_release --gh_token <gh_token> --prj_name [citus|citus_enterprise] --prj_ver <prj_ver> --schema_version <schema_version> [--cherry_pick_enabled --earliest_pr_date <Y.m.d>]
Arguments
--gh_token: Personal Access Token (PAT) for Github to be used
--prj_name: Project to be configured
--prj_ver: Version of the project to be released e.g. 10.2.0
--schema_version: schema version to be updated in configuration files
--cherry_pick_enabled: If used, the script cherry_picks all the PR's with backport label after earliest_pr_date
--earliest_pr_date: Used if cherry_pick_enabled is set. Start date for the PR's to be cherry-picked
This script is reponsible for updating version information in package configuration files
python -m tools.packaging_automation.update_package_properties --gh_token <gh-token> --prj_name [citus|citus_enterprise|pg_auto_failover|pg_auto_failover_enterprise] --tag_name <tag_name> [--fancy_version_no <version_no>] --email <email> --name <name> [--pipeline --exec_path <exec_path>]
Arguments
--gh_token: Personal Access Token (PAT) for Github to be used
--prj_name: Project to be updated.
--tag_name: Version to be released should be tagged in the project repo. This is the tag for the release. Should start with v e.g v10.2.0
--email: Email to be written in changelog and to author the PR
--name: Name to be written in changelog and to author the PR
--pipeline: If set, project code will not be cloned.
--exec_path: If pipeline is set, this parameter is mandatory. Path of project packaging code and configuration files
If you are using without pipeline parameter, packaging project code is cloned and at the end of the operation cloned code weill be deleted. If pipeline parameter is passed. then exec_path which is the path for the existing packaging code should be provided.
This script updates pgxn configuration files and creates a PR for changes to be reviewed.
python -m tools.packaging_automation.update_pgxn --gh_token <github_token> --prj_ver <prj_ver> [--pipeline --exec_path <exec_path>]
Arguments
--gh_token: Personal Access Token (PAT) for Github to be used
--prj_ver: Project version to be released. It is the same as tag except this parameter does not start with v e.g. 10.2.0
--pipeline: If set, project code will not be cloned.
--exec_path: If pipeline is set, this parameter is mandatory. Path of project packaging code and configuration files
If you are using without pipeline parameter, packaging project code is cloned and at the end of the operation cloned code weill be deleted. If pipeline parameter is passed. then exec_path which is the path for the existing packaging code should be provided.
This script creates rpm and deb packages for the given os and distro
python -m tools.packaging_automation.citus_package --gh_token <gh_token> --platform [el/7|ol/7|el/8|debian/stretch|debian/buster|debian/bullseye|ubuntu/bionic|ubuntu/focal] --build_type [release|nightly] --secret_key <packaging_secret_key> --passphrase <packaging_passphrase> --output_dir <output_dir> --input_files_dir <input_dir>
Arguments
--gh_token: Personal Access Token (PAT) for Github to be used
--platform: Platform of the package to be created
--build_type: Defines the package whethere release or nightly. Nightly packages are created from main branch, release packages are created from tag to be provided in pkgvars file in packaging files
--secret_key: gpg key to be used for signing packages after creating a local gpg key you can export as parameter and use it here export PACKAGING_SECRET_KEY=$(gpg --batch --export-secret-keys -a | base64)
--passphrase: passphrase for the secret key
--output_dir: path of the packages to be created
--input_files:dir: Path of the packaging files to be used for packaging
Uploads built packages into given packagecloud repository. If current branch is not equal to main branch, packages will not be pushed.
python -m tools.packaging_automation.upload_to_package_cloud --platform [el/7|ol/7|el/8|debian/stretch|debian/buster|debian/bullseye|ubuntu/bionic|ubuntu/focal] --package_cloud_api_token <package_cloud_api_token> --repository_name <repo_name> --output_file_path <output_path> --current_branch <current_branch> --main_branch <main_branch>
Arguments
--platform: Platform of the package to be created
--package_cloud_api_token: Api token for package cloud to be used for package push
--repository_name: repository name of the package to be published
--output_file_path: the path of the packages to be published
--current_branch: current branch that this script is being executed
--main_branch: main branch of the project to be checked
This script updates versions in Dockerfiles
python -m tools.packaging_automation.update_docker --gh_token <gh_token> --prj_ver <prj_ver>
Arguments
--gh_token: Personal Access Token (PAT) for Github to be used
--prj_ver: Project version to be released. It is the same as tag except this parameter does not start with v e.g. 10.2.0
--pipeline: If set, project code will not be cloned.
--exec_path: If pipeline is set, this parameter is mandatory. Path of project packaging code and configuration files
Builds and publishes docker images with the configured files with the update_docker script
Publish Docker on Branch Push
python -m tools.packaging_automation.publish_docker --pipeline_trigger_type push --github_ref refs/heads/<branch_name> --image_type [latest|alpine|nightly|postgres_12|postgres_13]
Publish Docker on Tag Push
python -m tools.packaging_automation.publish_docker --pipeline_trigger_type push --github_ref refs/tags/<tag_name> --image_type [latest|alpine|nightly|postgres_12|postgres_13]
Publish Docker Manually (Tags)
python -m tools.packaging_automation.publish_docker --pipeline_trigger_type workflow_dispatch --manual_trigger_type tags --github_ref refs/heads/master --tag_name <tag_name>
Publish Docker Manually (Main images)
python -m tools.packaging_automation.publish_docker --pipeline_trigger_type workflow_dispatch --manual_trigger_type main --github_ref refs/heads/master
Publish Docker Manually (Nightly images)
python -m tools.packaging_automation.publish_docker --pipeline_trigger_type workflow_dispatch --manual_trigger_type schedule --github_ref refs/heads/master
Arguments
--pipeline_trigger_type: Trigger type of the script which changes the script's flow and behaviour
--manuel_trigger_type: This argument is required when pipeline_trigger_type is set as workflow_dispatch. If set as tags, builds images and tags with tag name. Major minor and patch versions will be created as well
--tag_name: This argument is required when manual_trigger_type is set as tags.
--image_type: When pipeline_trigger_type is push, this argument is set to decide which image is built and published
--github_ref: This argument is set by Github actions which is in refs/heads/<branch_name> format in branch push and /refs/tags/<tag_name> in tag push
Branch name/tag name is extracted from this parameter and the extracted parameter is used to decide whether images published or not