diff --git a/.gitignore b/.gitignore index e69de29b..394d0449 100644 --- a/.gitignore +++ b/.gitignore @@ -0,0 +1,3 @@ +# Ignore everything in this directory +.yardoc/* + diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 00000000..f994c51b --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,15 @@ +version: 2 + +build: + os: "ubuntu-22.04" + tools: + python: "3.11" + +# Build from the docs/ directory with Sphinx +sphinx: + configuration: source/conf.py + +# Explicitly set the version of Python and its requirements +python: + install: + - requirements: requirements.txt \ No newline at end of file diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..a2e58bba --- /dev/null +++ b/Makefile @@ -0,0 +1,28 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# New target that executes the bash script before calling the original target +build-docs: pre-build # Change 'html' to the target you want to execute + +pre-build: + # Execute your bash script here + bash api-reference.sh + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +html: + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + diff --git a/requirements.in b/requirements.in new file mode 100644 index 00000000..da3e99a3 --- /dev/null +++ b/requirements.in @@ -0,0 +1,4 @@ +sphinxnotes-strike +sphinx +sphinx_rtd_theme +readthedocs-sphinx-search \ No newline at end of file diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00000000..ec263df8 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,62 @@ +# +# This file is autogenerated by pip-compile with Python 3.10 +# by the following command: +# +# pip-compile +# +alabaster==0.7.13 + # via sphinx +babel==2.12.1 + # via sphinx +certifi==2023.5.7 + # via requests +charset-normalizer==3.1.0 + # via requests +docutils==0.18.1 + # via + # sphinx + # sphinx-rtd-theme +idna==3.4 + # via requests +imagesize==1.4.1 + # via sphinx +jinja2==3.1.2 + # via sphinx +markupsafe==2.1.3 + # via jinja2 +packaging==23.1 + # via sphinx +pygments==2.15.1 + # via sphinx +readthedocs-sphinx-search==0.3.1 + # via -r requirements.in +requests==2.31.0 + # via sphinx +snowballstemmer==2.2.0 + # via sphinx +sphinx==6.2.1 + # via + # -r requirements.in + # sphinx-rtd-theme + # sphinxcontrib-jquery + # sphinxnotes-strike +sphinx-rtd-theme==1.2.2 + # via -r requirements.in +sphinxcontrib-applehelp==1.0.4 + # via sphinx +sphinxcontrib-devhelp==1.0.2 + # via sphinx +sphinxcontrib-htmlhelp==2.0.1 + # via sphinx +sphinxcontrib-jquery==4.1 + # via sphinx-rtd-theme +sphinxcontrib-jsmath==1.0.1 + # via sphinx +sphinxcontrib-qthelp==1.0.3 + # via sphinx +sphinxcontrib-serializinghtml==1.1.5 + # via sphinx +sphinxnotes-strike==1.2 + # via -r requirements.in +urllib3==2.0.3 + # via requests diff --git a/source/citus-docker-images.rst b/source/citus-docker-images.rst new file mode 100644 index 00000000..d7708aff --- /dev/null +++ b/source/citus-docker-images.rst @@ -0,0 +1,38 @@ +Citus Docker images +==================== + +General Structure +------------------ +Docker images are part of the Citus release process. They are built and pushed after every release. Dockerfiles are stored in the `citusdata/docker `_ repository. In each release, we change the below Dockerfiles to point to the new Citus release. +Citus Docker images are published in DockerHub `citusdata/citus `_ + +1. `Main Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version. +2. `alpine Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version with the alpine base Docker image. +3. `postgres-xx` Dockerfiles (where `xx` represents the major Postgres versions) : Used to build the Citus images for the older Postgres versions. We normally support the last 3 major Postgres versions. For example, if the latest Postgres version is 15, we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported Postgres major versions other than the latest one. + +Build Process +------------------ +The Dockerfiles are built and pushed to Docker Hub by GH Actions pipelines. There are four pipelines used for the Citus Docker image build and publish process: + +1. `Update Citus versions `_: This pipeline updates the Citus versions in the Dockerfiles and pushes the changes to a newly created branch and creates a PR. It should be triggered manually after finishing the publishing of Citus packages to the packagecloud. + +2. `Build and publish Citus Docker images on push `_: This pipeline is triggered on every branch push to the repository. It builds the Docker images on every occasion and pushes them to Docker Hub if the branch is master. + +3. `Build and publish Citus Docker images on schedule `_: This pipeline is triggered every day at 00:30 UTC and builds and publishes the Docker images for the main, alpine, and `postgres-xx` Docker images to Docker Hub. + +4. `Build and publish Citus Docker images manually `_: This pipeline is triggered manually and builds and publishes the Docker images. It is useful when we want to build and publish the Docker images without waiting for the scheduled pipeline to run. + +New Postgres Version Support +----------------------------- +When a new Postgres version is released, we need to add support for the new Postgres version in the Dockerfiles. To add support for a new Postgres version: + +1. Create a new branch from the master branch. +2. Create a new Dockerfile for the new Postgres version. For example, for Postgres 16, create `postgres-16/Dockerfile` by copying the `postgres-15/Dockerfile`. +3. Change the Postgres version in the new Dockerfile to the new Postgres version. +4. Test the new Dockerfile using the appropriate Docker build and run commands. +5. Add new Postgres version support to the scheduled pipeline. +6. If the tests are successful, create a pull request to merge the changes to the master branch. + +Adding new Postgres version support to the tools scripts +--------------------------------------------------------- +When a new Postgres version is released, we also need to add support for the new Postgres version in the tools scripts. Mainly, the `update_docker.py `_ script needs to be updated to add support for the new Postgres version. This script is used in the pipelines. diff --git a/source/conf.py b/source/conf.py new file mode 100644 index 00000000..b615f09c --- /dev/null +++ b/source/conf.py @@ -0,0 +1,31 @@ +from datetime import date + +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = "Citus Packaging" +copyright = f"{date.today().year} .Citus Data Licensed under the MIT license, see License for details. " +author = "Citus Data" +release = "2.2.0" + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = ["sphinxnotes.strike"] + +templates_path = ["_templates"] +exclude_patterns = [] + +language = "python" + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = "sphinx_rtd_theme" +html_static_path = ["_static"] + diff --git a/source/debian-packages.rst b/source/debian-packages.rst new file mode 100644 index 00000000..4be76e8b --- /dev/null +++ b/source/debian-packages.rst @@ -0,0 +1,57 @@ +.. _debian-packages: + +Debian Packages +=============== + +Introduction to Debian packaging +-------------------------------- + +Debian packaging provides a standardized way to create software packages for Debian-based operating systems. It ensures easy installation, upgrade, and removal of software components. The packaging format used in Debian is known as the Debian package format or ``.deb``. + +Debian-specific tools and conventions +------------------------------------- + +Debian packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following: + +- **dpkg**: The Debian package manager (``dpkg``) is the core tool for installing, querying, and managing Debian packages. It operates at a lower level than higher-level package management tools like ``apt`` or ``apt-get``. + +- **Debian control file**: The ``debian/control`` file is a metadata file that provides information about the package, its dependencies, and other package-related details. It includes fields such as package name, version, maintainer, description, dependencies, and more. Maintaining an accurate and up-to-date control file is essential for proper package management. + +- **Debian changelog**: The ``debian/changelog`` file maintains a record of changes made to the package across different versions. It includes information about the package version, the person responsible for the change, the change description, and other relevant details. The changelog helps users and maintainers track the history of the package and understand the modifications introduced in each release. + +- **Debian rules file**: The ``debian/rules`` file is a makefile-like script that defines the build process and other package-specific tasks. It specifies how to compile the source code, configure the package, and install the files in the correct locations. The rules file allows for customization and automation of the packaging process. + +Other important Debian-specific tools and files include ``debuild``, ``lintian``, and ``pbuilder``, which aid in building, linting, and testing packages respectively. + +Additional Debian Files +----------------------- + +In addition to the core Debian packaging files, the packaging process may involve other files with specific purposes. Here are some of the additional Debian files commonly used: + +- **Source/format**: This file specifies the format version of the Debian source package. + +- **Source/lintian-overrides**: This file is used to override specific lintian warnings or errors generated during the package build process. + +- **Tests/control**: This file contains information about the tests to be run on the package. + +- **Tests/installcheck**: This file specifies commands or scripts to be executed after the package is installed. + +- **Upstream/signing-key.asc**: This file contains the ASCII-armored representation of the upstream signing key used to verify the authenticity and integrity of the package. + +- **Changelog**: The ``debian/changelog`` file maintains a record of changes made to the package. + +- **Compat**: The ``debian/compat`` file specifies the compatibility level of the package's packaging format. + +- **Control.in**: The ``debian/control.in`` file is a template file used to generate the ``debian/control`` file. + +- **Copyright**: The ``debian/copyright`` file provides information about the copyright holders and licensing terms. + +- **Docs**: The ``debian/docs`` file lists the additional documentation files included with the package. + +- **NOTICE**: The ``debian/NOTICE`` file contains any legally required notices or attributions. + +- **Pgversion**: The ``debian/pgversion`` file specifies the PostgreSQL version for which the package is intended. + +- **Rules**: The ``debian/rules`` file is a makefile-like script that defines the build process. + +For more information about Debian packaging, see the `Debian New Maintainers' Guide `_. diff --git a/source/index.rst b/source/index.rst new file mode 100644 index 00000000..cf8dc9fb --- /dev/null +++ b/source/index.rst @@ -0,0 +1,75 @@ + +Welcome to Citus Data Packaging documentation! +====================================================== + +|Citus Linux Packages Nightly Status| |Citus Docker Image Nightly Status| |Latest Documentation Status| + +.. |Citus Linux Packages Nightly Status| image:: https://github.com/citusdata/packaging/actions/workflows/build-citus-community-nightlies.yml/badge.svg + :target: https://github.com/citusdata/packaging/actions/workflows/build-citus-community-nightlies.yml + :alt: Citus Linux Packages Nightly Status + +.. |Citus Docker Image Nightly Status| image:: https://github.com/citusdata/docker/actions/workflows/publish_docker_images_cron.yml/badge.svg + :target: https://github.com/citusdata/docker/actions/workflows/publish_docker_images_cron.yml + :alt: Citus Docker Image Nightly Status + + +.. |Latest Documentation Status| image:: https://readthedocs.org/projects/django-multitenant/badge/?version=latest + :target: https://citusdata-packaging.readthedocs.io/en/latest//?badge=latest + :alt: Documentation Status + + +Citus Data Packaging Documentation +======================================== + +Welcome to the official documentation for ``CitusData Packaging``. + +``CitusData Packaging`` is a collection of scripts and tools that are used to packaging Citus Data products. +This documentation is intended to provide a detailed overview of the packaging process and the tools used to package Citus Data products. + + + +.Table of Contents +=================== + + +.. toctree:: + :maxdepth: 2 + + introduction + + +.. toctree:: + :maxdepth: 2 + :caption: Repository Management + + package-build-environment + repository-installation-scripts + repository + + +.. toctree:: + :maxdepth: 2 + :caption: Tools and Configurations + + tools-scripts + project-packaging-configurations + + +.. toctree:: + :maxdepth: 2 + :caption: Artifacts + + debian-packages + rpm-packages + citus-docker-images + pgdg + pgxn + + +.. toctree:: + :maxdepth: 2 + :caption: Packaging Process + + packaging-process + + diff --git a/source/introduction.rst b/source/introduction.rst new file mode 100644 index 00000000..d0018190 --- /dev/null +++ b/source/introduction.rst @@ -0,0 +1,44 @@ +Introduction +============ + +Welcome to the documentation outlining the packaging process for Citus and its associated projects. Packaging plays a crucial role in the distribution and deployment of software, ensuring that it can be easily installed and utilized across various operating systems and environments. + +In this document, we will explore the steps and best practices involved in preparing packages for different operating systems and releases. We will focus on the Debian and RPM package formats, as they are widely used and supported in the industry. Additionally, we will cover the creation and management of Docker environments to facilitate packaging for specific OS/release/PostgreSQL version combinations. + +The supported operating systems and releases for packaging include: + +- CentOS 8 +- CentOS 7 +- Oracle Linux 8 +- Oracle Linux 7 +- AlmaLinux 9 +- Debian Buster +- Debian Bullseye +- Debian Bookworm +- Ubuntu Bionic +- Ubuntu Focal +- Ubuntu Jammy +- Ubuntu Kinetic + +The packaging process will involve preparing packages for various projects, including: + +* `Citus `_ +* `Pgazure `_ +* Cron + * `Cron Debian `_ + * `Cron RPM `_ +* Hll + * `Hll Debian `_ + * `Hll RPM `_ +* Topn + * `Topn Debian `_ + * `Topn RPM `_ +* `Azure-gdpr `_ + +Once the packages are ready, we will push them to PackageCloud, a package management and distribution service. Additionally, we will create Docker images for each OS/release/PostgreSQL version combination, which will be published on Docker Hub. + +Lastly, we will cover the publishing of the applications into PGXN (PostgreSQL Extension Network), a platform that hosts and distributes PostgreSQL extensions. + +By following the steps outlined in this document, you will gain a comprehensive understanding of the packaging process for Citus and related projects. This knowledge will empower you to efficiently distribute and deploy the software across a wide range of operating systems and environments. + +Now, let's dive into the detailed steps for package preparation, Docker environment management, Docker image creation, PackageCloud integration, and PGXN publishing. diff --git a/source/package-build-environment.rst b/source/package-build-environment.rst new file mode 100644 index 00000000..9176e86b --- /dev/null +++ b/source/package-build-environment.rst @@ -0,0 +1,107 @@ +Packaging Build Environment Management +====================================== + +Introduction to Packaging Environments +-------------------------------------- + +In our packaging process, we utilize Docker images for each OS/release/PostgreSQL version combination. Docker images offer a reliable and consistent environment for packaging software applications. They provide several advantages for the packaging process: + +1. **Isolation**: Docker containers encapsulate applications and their dependencies, ensuring isolation from the underlying host system. This isolation helps prevent conflicts between different software components and provides a clean and reproducible environment for packaging. + +2. **Reproducibility**: Docker environments enable the creation of reproducible packaging environments. By defining the dependencies and configurations within a Docker image, it becomes easier to recreate the same environment across different systems, ensuring consistent packaging results. + +3. **Portability**: Docker environments are portable and can be easily shared across different systems. This portability ensures that the same packaging results can be achieved across different systems, eliminating environment-related issues. + +4. **Scalability**: Docker environments can be easily scaled up or down to meet the packaging requirements. This scalability ensures that the packaging process can be efficiently adjusted to handle different workload sizes. + +5. **Other Benefits**: Docker environments provide a secure, reliable, consistent, flexible, efficient, simple, and maintainable packaging environment. Docker isolates the packaging process from the underlying host system, ensuring a controlled and stable environment. + +Docker Images for Various Package Types +--------------------------------------- + +We utilize Docker images for each OS/release/PostgreSQL version combination to create a reproducible and consistent packaging environment. + +For Debian-based packages, we can use the same Docker image for all PostgreSQL versions. + +For example, the Debian/10 Docker image can be used for both PostgreSQL 13 and PostgreSQL 14 packages. + +However, for RPM-based packages, we need a dedicated Docker image for each PostgreSQL version. + +For instance, we require a Docker image for CentOS/7 with PostgreSQL 13 and another Docker image for CentOS/7 with PostgreSQL 14 packages. + +Here are the supported OS/release pairs: + +1. CentOS + - centos/7 + +2. Oracle Linux + - oraclelinux/7 + - oraclelinux/8 + +3. AlmaLinux + - almalinux/9 + +4. Debian + - debian/buster + - debian/bullseye + - debian/bookworm + +5. Ubuntu + - ubuntu/bionic + - ubuntu/focal + - ubuntu/jammy + - ubuntu/kinetic + +6. PGXN + +Based on these OS/release pairs, the Docker images for each OS/release/PostgreSQL version combination are as follows: + +1. CentOS + - centos/7 PostgreSQL 13 + - centos/7 PostgreSQL 14 + - centos/7 PostgreSQL 15 + - centos/7 PostgreSQL 16 (planned) +2. Oracle Linux + - oraclelinux/7 PostgreSQL 13 + - oraclelinux/7 PostgreSQL 14 + - oraclelinux/7 PostgreSQL 15 + - oraclelinux/7 PostgreSQL 16 (planned) + - oraclelinux/8 PostgreSQL 13 + - oraclelinux/8 PostgreSQL 14 + - oraclelinux/8 PostgreSQL 15 + - oraclelinux/8 PostgreSQL 16 (planned) +3. AlmaLinux + - almalinux/9 PostgreSQL 13 + - almalinux/9 PostgreSQL 14 + - almalinux/9 PostgreSQL 15 + - almalinux/9 PostgreSQL 16 (planned) +4. Debian + - debian/buster (all PostgreSQL versions) + - debian/bullseye (all PostgreSQL versions) + - debian/bookworm (all PostgreSQL versions) +5. Ubuntu + - ubuntu/bionic (all PostgreSQL versions) + - ubuntu/focal (all PostgreSQL versions) + - ubuntu/jammy (all PostgreSQL versions) + - ubuntu/kinetic (all PostgreSQL versions) +6. PGXN + +Developing and Maintaining Docker Images +----------------------------------------- + +The Dockerfiles used to build Docker images are located in the "develop" branch of the packaging repository. We follow a template structure to auto-generate Docker files for each OS/release/PostgreSQL version combination. The templates used are: + +1. `Dockerfile-deb.tmpl `_: This template generates Docker files for Debian-based packages. +2. `Dockerfile-rpm.tmpl `_: This template generates Docker files for RPM-based packages. +3. `Dockerfile-pgxn.tmpl `_: This template generates Docker files for PGXN-based packages. + +After making changes to the template files, the `update_dockerfiles.sh `_ script needs to be run to generate the Docker files for each OS/release/PostgreSQL version combination. + +Once the changes are committed and pushed, the GitHub Actions workflow will build the Docker images using the generated Docker files. + +After confirming that all Docker images are successfully built, the changes can be merged into the master branch. + +When the changes are merged into the master branch, the GitHub Actions workflow will push the Docker images to the `citus/packaging `_ Docker Hub repository. + +If you want to publish test images from `citus/packaging-test `_, you can use the `test pipeline `_. To use the test pipeline, change the current branch to your branch name. The test pipeline will then push the Docker images to the `citus/packaging-test `_ Docker Hub repository. + diff --git a/source/packaging-process.rst b/source/packaging-process.rst new file mode 100644 index 00000000..fd6db974 --- /dev/null +++ b/source/packaging-process.rst @@ -0,0 +1,143 @@ +Packaging Process +================= + +The packaging process consists of the following steps: + +1. Bake deb and rpm packages +2. Bake docker images +3. Bake pgxn package +4. Open issue for PGDG packaging + +Bake deb and rpm packages +--------------------------- + +In this phase, the packaging configuration files need to be edited to bake the packages for the desired application version. + +Editing Configuration Files by Pipeline +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Citus has an automation process to edit the packaging configuration scripts called the `Update Package Properties `_ workflow. + +By executing this workflow on the all-citus branch, a new pull request (PR) will be generated in the packaging repository. + +This PR will update the packaging configuration files to the desired version. + +Once the necessary checks have passed, a review can be requested from the packaging team. + +After the review is completed, the PR can be merged. + +Once all the jobs are finished, the new packages can be found in the `Citusdata Package Repository `_. + +It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. + +Editing Configuration Files Manually +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For projects like pg_azure_storage and azure-gdpr that do not have automation pipelines, the configuration files need to be edited manually. +There are two categories of projects that require manual configuration file editing: + +1. Projects with deb and rpm configuration files in the same branch: + - For projects in this category, such as pg_azure_storage and azure-gdpr, the following steps can be followed: + - Create a new branch from the latest commit on the master branch. + - Edit the configuration files to the desired version: + - Add a new entry to the `debian/changelog` file. + - Update the version in the pkgvars file. + - Update the version in the rpm spec file. + - Add a new changelog entry to the rpm spec file. + - Create a pull request (PR) from the new branch to the master branch. + - After the PR is merged, the new packages can be found in the `Citusdata Package Repository `_. It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. + +2. Projects with deb and rpm configuration files in different branches: + - For projects like topn, hll, and cron that have deb and rpm configuration files in different branches, the following steps can be followed: + + **Debian Configuration Files:** + - Checkout the debian- branch. + - Create a new branch from the latest commit on the debian- branch. + - Edit the configuration files to the desired version: + + - Add a new entry to the `debian/changelog` file. + - Update the version in the pkgvars file. + + - Create a PR from the new branch to the debian- branch. + - After the PR is merged, the new packages can be found in the `Citusdata Package Repository `_. + It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. + + **Redhat Configuration Files:** + - Checkout the rpm- branch. + - Create a new branch from the latest commit on the rpm- branch. + - Edit the configuration files to the desired version: + + - Update the version in the rpm spec file. + - Add a new changelog entry to the rpm spec file. + + - Create a PR from the new branch to the rpm- branch. + - After the PR is merged, the new packages can be found in the `Citusdata Package Repository `_. It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. + +Bake docker images +------------------ + +This step is applicable only for Citus Community. + +Baking Main versions +~~~~~~~~~~~~~~~~~~~~ + +In this phase, the docker image configuration files need to be edited to bake the docker images for the desired application version. + +The `Update Version on Docker Files `_ workflow can be used to edit the docker image configuration files. + +By executing this workflow on the master branch, a new pull request (PR) will be generated in the docker repository. + +This PR will update the docker image configuration files to the desired version. + +Once all the necessary checks have passed, a review can be requested from the packaging team. + +After the review is completed, the PR can be merged. + +Once all the jobs are finished, the new docker images can be found in the `Citusdata Docker Repository `_. + +The versioned images should appear in the docker repository for all supported postgres versions and alpine. + +Create Images for Patch Releases +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In this phase, docker images need to be created for patch releases. +For example, if there is a patch release for version 10.1.1, docker images need to be created for versions 10, 10.1, and 10.1.1. + +To create docker images for patch releases, a tag needs to be added to the latest commit on the master branch and pushed. + +The `The tag pipeline `_ pipeline will create the docker images for the tag and push them to the docker repository. + +Once all the jobs are finished, the new docker images can be found in the `Citusdata Docker Repository `_. + +The versioned images should appear in the docker repository for all supported postgres versions and alpine. + +Bake pgxn package +----------------- + +When building the pgxn extension for Citus, the following steps need to be followed: + +1. Execute the `Update Version on PGXN Config Files `_ pipeline to update the version in the pgxn configuration files. This pipeline will create a pull request (PR) with the updated version. +2. Check for the minimum postgres version and update it if necessary. +3. Check that all the tests pass and notify the PR reviewer to review the PR. +4. Once the PR is reviewed, merge the PR and ensure that the Citus PGXN is updated with the new version. + +Opening an issue for PGDG packaging +----------------------------------- + +After all the packages are baked, an issue needs to be opened for PGDG packaging. PGDG is a repository of PostgreSQL packages for several Linux distributions. + +Citus packages are available for RPM-based distributions. +To request a new release for Citus, an issue needs to be opened in the `Postgres Redmine `_. + +The required fields for the issue are as follows: + +- Tracker: Bug +- Subject: New release of Citus +- Description: Provide a detailed description of the new release, including any notable changes or improvements. +- Category: Packaging +- Priority: Normal +- Target version: +- Assignee: Leave blank unless specified + +Please make sure to include all the necessary information and follow the bug reporting guidelines provided by the Postgres Redmine. + diff --git a/source/pgdg.rst b/source/pgdg.rst new file mode 100644 index 00000000..66f2db0f --- /dev/null +++ b/source/pgdg.rst @@ -0,0 +1,8 @@ +Citus PGDG Packages +==================== + +Citus packages for RPM-based distributions are made available through the PGDG (PostgreSQL Global Development Group) repository. +PGDG is a repository that provides PostgreSQL packages for various Linux distributions. + +It is important to note that the packaging process for Citus packages in PGDG is not directly managed by us. +Instead, we open an issue in the PGDG repository and Devrim Gunduz, the owner of PGDG, takes care of the packaging process. diff --git a/source/pgxn.rst b/source/pgxn.rst new file mode 100644 index 00000000..3176d4e1 --- /dev/null +++ b/source/pgxn.rst @@ -0,0 +1,25 @@ +Citus PGXN Extension Builds +=========================== + +PGXN Build Infrastructure +---------------------------- +In the packaging system, we have scripts and images to build PGXN extensions. +In the package build environment, we have Docker images, specifically Dockerfiles, to build pgxn extensions. + +The `Docker image `_ serves as the base image for all pgxn extensions. It contains all the dependencies required for building pgxn extensions. +Within this docker image, there is a script called `fetch_and_build_pgxn `_ which is used to build pgxn extensions. + +Citus PGXN Configuration +------------------------ + +For each project package configuration, we maintain a branch named `pgxn-citus `_ specifically for building the pgxn extension for Citus. In this branch, we have three configuration files used for building the packages: + +1. `META.json `_ + +2. pkgvars + +3. postgres-matrix.yml + +The usage of pkgvars and postgres-matrix.yml is similar to other project builds. + +.. _Pgxn Build Process: diff --git a/source/project-packaging-configurations.rst b/source/project-packaging-configurations.rst new file mode 100644 index 00000000..4def845b --- /dev/null +++ b/source/project-packaging-configurations.rst @@ -0,0 +1,65 @@ +Project Packaging Configurations +================================ + +For each project, a new branch is created in the packaging repository where packaging configuration files for the project are added. +The packaging configuration files include: + +1. Pkgvars +2. Postgres-matrix.yml + +Pkgvars +------- +Pkgvars is a property file that contains essential information about the package. +It is used by the packaging system to generate the package name, version, and other details. + +Here is an example of a pkgvars file: + +.. code-block:: ini + + pkgname=citus + hubproj=citus + pkgdesc='Citus (Open-Source)' + pkglatest=11.3.0.citus-1 + nightlyref=main + versioning=fancy + +Parameters for the pkgvars file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1. pkgname: Prefix of the package. +2. hubproj: Name of the repository in GitHub. +3. pkgdesc: Description of the package. It is for reference purposes and not used by the packaging system. +4. pkglatest: Latest version of the package. It is used by the packaging system to generate the package name and retrieve the release number from the GitHub repository. +5. nightlyref: Branch name of the nightly builds. +6. versioning: Versioning scheme. It can be either 'fancy' or 'simple'. If 'fancy', fancy version numbers are added to the package name. The default is 'simple'. + +Postgres-matrix.yml +------------------- +The Postgres-matrix.yml file contains the Citus Postgres version matrix. +The packaging system utilizes this file to determine the appropriate Postgres versions for a given Citus version. + +Here is an example of a postgres-matrix.yml file: + +.. code-block:: yaml + + name: Postgres Version Matrix + project_name: citus + version_matrix: + - 8.0: + postgres_versions: [10, 11] + - 9.0: + postgres_versions: [11, 12] + - 9.5: + postgres_versions: [11, 12, 13] + - 10.1: + postgres_versions: [12, 13] + - 10.2: + postgres_versions: [12, 13, 14] + - 11.0: + postgres_versions: [13, 14] + - 11.1: + postgres_versions: [13, 14, 15] + +In the packaging system, the version number is obtained from the pkgvars file, and the corresponding matrix entry is determined by comparing the version number with the entries in the matrix file. +The Postgres versions specified in the matrix entry are then used to generate individual packages. + +For instance, if we want to create a package for Citus version 9.3.8, we will retrieve the matrix entry for version 9.0 and generate a package for each Postgres version listed in the matrix entry since 9.3.8 falls between 9.0 and 9.5. diff --git a/source/repository-installation-scripts.rst b/source/repository-installation-scripts.rst new file mode 100644 index 00000000..ec536553 --- /dev/null +++ b/source/repository-installation-scripts.rst @@ -0,0 +1,61 @@ +Repository Installation Scripts +=============================== + +Citus packages are hosted on packagecloud.io. You can install the packages directly using the packagecloud.io installation scripts. +However, we also provide custom installation scripts for convenience. +The benefits of using the custom installation scripts are: + +1. The repository URL `install.citusdata.com/repos.citusdata.com` is used instead of packagecloud.io. + These URLs are managed by us, allowing isolation from potential changes to the packagecloud.io URL in the future. + +2. The PostgreSQL repository is automatically installed alongside our packages for ease of use. + +3. Essential programs such as curl, wget, gnupg, and apt-transport-https are installed if not already present, + as they are required for installing PostgreSQL and Citus packages. + +The Repository Installation Scripts are located in the `packaging/gh-pages branch `_. +These files are served using GH Pages, and the GH Pages settings can be found in the `settings `_. +The original URL prefix for the files is `https://citusdata.github.io/packaging/`. +For example, the URL for `deb.sh `_. +`install.citusdata.com/repos.citusdata.com` is a CNAME record for `citusdata.github.io` in our DNS settings on Cloudflare. +For more information about URL redirection, please refer to the Citus Packaging Web Url Certificates document. + +There are 8 scripts in this branch for 4 repositories on packagecloud.io: + +1. community + + - `Community deb.sh `_ + + - `Community rpm.sh `_ + +2. community-nightlies + + - `community-nightlies deb.sh `_ + + - `community-nightlies rpm.sh `_ + +3. enterprise + + - `enterprise deb.sh `_ + + - `enterprise rpm.sh `_ + +4. enterprise-nightlies + + - `enterprise-nightlies deb.sh `_ + + - `enterprise-nightlies rpm.sh `_ + +These scripts are used to install the corresponding repositories. For example, to install the community repository, run the following command: + +For Debian/Ubuntu: + +.. code-block:: shell + + curl https://install.citusdata.com/community/deb.sh | sudo bash + +For RHEL/CentOS: + +.. code-block:: shell + + curl https://install.citusdata.com/community/rpm.sh | sudo bash diff --git a/source/repository.rst b/source/repository.rst new file mode 100644 index 00000000..c45dd7e2 --- /dev/null +++ b/source/repository.rst @@ -0,0 +1,9 @@ +Package Repositories +===================== + +Baked packages are stored in various repositories. The repositories are: + +* `PackageCloud `_: Stores deb and rpm packages. Community users and Marlin uses this repository. This is the main repo for all users +* PGDG: Stores only latest Citus rpm packages. You can not find older versions here. We do not manage the packages here directly. We open a ticket with PGDG to create a new package. +* `PGXN `_: The PostgreSQL Extension Network (PGXN) is a community-driven effort aimed at providing a centralized platform for distributing and sharing extensions for the PostgreSQL database management system. We use this repository to store the citus extension. +* `Docker Hub `_: Stores docker images. We use this repository to store the docker images for citus. \ No newline at end of file diff --git a/source/rpm-packages.rst b/source/rpm-packages.rst new file mode 100644 index 00000000..47a1bb04 --- /dev/null +++ b/source/rpm-packages.rst @@ -0,0 +1,30 @@ +RPM Packages +------------ + +Introduction to RPM packaging +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +RPM packaging is a common method used to create software packages for RPM-based Linux distributions. RPM (Red Hat Package Manager) provides a reliable and consistent way to install, upgrade, and remove software components. The packaging format used in RPM is known as the RPM package format or `.rpm`. + +RPM-specific tools and conventions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +RPM packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following: + +- **rpmbuild**: The `rpmbuild` command is the primary tool for building RPM packages. It compiles the source code, creates the package structure, and generates the final RPM package. + +- **.spec file**: The `.spec` file is a key component of RPM packaging. It contains detailed instructions and metadata about the package, such as its name, version, release, dependencies, build requirements, and file locations. The `.spec` file defines the build process, file ownership, permissions, and other package-specific configurations. + +- **rpmlintrc**: The `rpmlintrc` file is used to configure the rpmlint tool, which performs static analysis on the generated RPM package. The rpmlint tool checks for common packaging issues, compliance with guidelines, and potential errors. The `rpmlintrc` file allows you to specify specific checks to enable or disable for your package. + +Additional RPM Files +~~~~~~~~~~~~~~~~~~~~ + +In addition to the core RPM packaging components, the packaging process may involve other files with specific purposes. Here are some additional RPM-related files commonly used: + +- **.spec file**: The `.spec` file contains the build instructions and metadata for the RPM package. + +- **rpmlintrc**: The `rpmlintrc` file configures the rpmlint tool for checking the package. + +For more information about RPM packaging, see the [RPM Packaging Guide](https://rpm-packaging-guide.github.io/). + diff --git a/source/tools-scripts.rst b/source/tools-scripts.rst new file mode 100644 index 00000000..b6b66696 --- /dev/null +++ b/source/tools-scripts.rst @@ -0,0 +1,28 @@ +Packaging Tools Scripts +======================= + +Packaging operations are executed using a collection of Python scripts stored in the `tools `_ repository. +These scripts are written in Python and require Python 3.8 or higher. They have been tested on Ubuntu focal. + +Whenever a change is made, a tag is added to the commit and pushed to the repository. The tag follows the `v..` format. +The scripts are primarily utilized within GH Actions pipelines, where the corresponding tag is used. + +The following scripts are employed for packaging purposes: + +Tools Scripts +------------- + +We utilize specific Python scripts to perform packaging operations. These scripts can be found at `https://github.com/citusdata/tools`. + +For each change, we create a new tag for the latest release and incorporate that tag into the pipelines. + +Packaging Scripts +~~~~~~~~~~~~~~~~~ + +The following provides an overview of the scripts used within the pipelines to execute various packaging operations: + +* citus_package: This is the main script responsible for building the packages. +* update_docker: This script generates Citus Docker images for end users. +* upload_to_package_cloud: Publishes the packages created by citus_package to Package Cloud. +* publish_docker: Publishes the Docker images created by the update_docker script. +* delete_packages_on_package_cloud: Removes nightly packages older than 10 days from Package Cloud to reduce repository size and lower Package Cloud bills.