From 30928d2beb52a8ae3d2d74f573ee2254bb29d56a Mon Sep 17 00:00:00 2001 From: gindibay Date: Mon, 26 Jun 2023 23:02:17 +0300 Subject: [PATCH 01/11] Adds initial files --- .gitignore | 3 + Makefile | 28 ++++++ requirements.in | 4 + source/citus-docker-images.rst | 79 +++++++++++++++++ source/conf.py | 30 +++++++ source/debian-packages.rst | 59 +++++++++++++ source/index.rst | 40 +++++++++ source/introduction.rst | 47 ++++++++++ source/package-build-environment.rst | 97 +++++++++++++++++++++ source/pgdg.rst | 15 ++++ source/pgxn.rst | 35 ++++++++ source/project-packaging-configurations.rst | 69 +++++++++++++++ source/repository-installation-scripts.rst | 47 ++++++++++ source/rpm-packages.rst | 30 +++++++ source/tools-scripts.rst | 10 +++ 15 files changed, 593 insertions(+) create mode 100644 Makefile create mode 100644 requirements.in create mode 100644 source/citus-docker-images.rst create mode 100644 source/conf.py create mode 100644 source/debian-packages.rst create mode 100644 source/index.rst create mode 100644 source/introduction.rst create mode 100644 source/package-build-environment.rst create mode 100644 source/pgdg.rst create mode 100644 source/pgxn.rst create mode 100644 source/project-packaging-configurations.rst create mode 100644 source/repository-installation-scripts.rst create mode 100644 source/rpm-packages.rst create mode 100644 source/tools-scripts.rst diff --git a/.gitignore b/.gitignore index e69de29bb..394d0449f 100644 --- a/.gitignore +++ b/.gitignore @@ -0,0 +1,3 @@ +# Ignore everything in this directory +.yardoc/* + diff --git a/Makefile b/Makefile new file mode 100644 index 000000000..a2e58bba9 --- /dev/null +++ b/Makefile @@ -0,0 +1,28 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# New target that executes the bash script before calling the original target +build-docs: pre-build # Change 'html' to the target you want to execute + +pre-build: + # Execute your bash script here + bash api-reference.sh + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +html: + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + diff --git a/requirements.in b/requirements.in new file mode 100644 index 000000000..da3e99a39 --- /dev/null +++ b/requirements.in @@ -0,0 +1,4 @@ +sphinxnotes-strike +sphinx +sphinx_rtd_theme +readthedocs-sphinx-search \ No newline at end of file diff --git a/source/citus-docker-images.rst b/source/citus-docker-images.rst new file mode 100644 index 000000000..48b1f7f60 --- /dev/null +++ b/source/citus-docker-images.rst @@ -0,0 +1,79 @@ +Citus Docker images +==================== + +General Structure +------------------ + +Docker images are part of the Citus release process. They are built and pushed after every release. +Dockerfiles are stored in `citusdata/docker `_ repository. +In each release, we change the below Dockerfiles to point to the new Citus release. +1. `Main Dockerfile `_ +which is used to build the latest regular Citus image for the latest Postgres version. +2. `alpine Dockerfile `_ +which is used to build the latest regular Citus image for the latest Postgres version +with the alpine base docker image. +3. `postgres-xx Dockerfiles which are used to build the Citus images for the older Postgres versions. +Normally, we support the last 3 major Postgres versions. For example, if the latest Postgres version is 15, +we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported postgres major versions other than the latest one. +For example, if we support 13, 14, and 15, we need to have postgres-13/Dockerfile +and postgres-14/Dockerfile Dockerfiles in our repository. + +Build Process +------------------ + +The Dockerfiles are built and pushed to Docker Hub by GH Actions pipelines. +There are two pipelines to be used for Citus docker image build and publish process. +1. `Update Citus versions `_: +This pipeline should be triggered manually after finishing publishing of Citus packages to the packagecloud, +since the pipeline uses the Citus packages from the packagecloud to build the docker images. +This pipeline updates the Citus versions in the Dockerfiles and pushes the changes +to a newly created branch and creates a PR. +2. `Build and publish Citus docker images on push `_: +This pipeline is triggered on every branch push to repository. +This pipeline builds the docker images on every occasion and pushes them to Docker Hub if the branch is master. +Therefore, this pipeline is used to validate the build in the branches other than master and to publish the images to Docker Hub in the master branch. +3. `Build and publish Citus docker images on schedule `_: +This pipeline is triggered every day at 00:30 UTC and build and publish the docker images for +main, alpine and postgres-xx docker images to Docker Hub. +4. `Build and publish Citus docker images manually `_: +This pipeline is triggered manually and build and publish the docker images. It is useful when we want to build and publish the docker images +without waiting for the scheduled pipeline to run. + +New Postgres Version Support +------------------ + +When a new Postgres version is released, we need to add the new Dockerfile for the new Postgres version. +Steps to add a new Postgres version support: +1. Create a new branch from master branch. +2. Create a new Dockerfile for the new Postgres version. For example for postgres 16 we need to add postgres-16/Dockerfile by copying the postgres-15/Dockerfile. +3. Change the postgres version in the new Dockerfile to the new Postgres version. +4. Test the new Dockerfile by running the below command in the root directory of the repository. + +``` +docker build -t citusdata/citus:16.0.0-postgres-16 . +``` +5. If the build is successful, test the new docker image by running the below command. + +``` +# run PostgreSQL with Citus on port 5500 +docker run -d --name citus -p 5500:5432 -e POSTGRES_PASSWORD=mypassword citusdata/citus:16.0.0-postgres-16 +# connect using psql within the Docker container +docker exec -it citus psql -U postgres +# or, connect using local psql +psql -U postgres -d postgres -h localhost -p 5500 +``` +6. Add new postgres version support to the scheduled pipeline by adding the new postgres version to the pipelines. +7. If the tests are successful, create a PR to merge the changes to master branch. + +Adding new postgres version support to the tools scripts +-------------------------------------------------------- + +When a new Postgres version is released, we need to add the new Postgres version support to the tools scripts. +Mainly `update_docker.py script +needs to be updated to add the new Postgres version support. +This script is being used in the pipelines. + + + + + diff --git a/source/conf.py b/source/conf.py new file mode 100644 index 000000000..a526b2e94 --- /dev/null +++ b/source/conf.py @@ -0,0 +1,30 @@ +from datetime import date + +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = "Citus Packaging" +copyright = f"{date.today().year} .Citus Data Licensed under the MIT license, see License for details. " +author = "Citus Data" +release = "2.2.0" + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = ["sphinxnotes.strike"] + +templates_path = ["_templates"] +exclude_patterns = [] + +language = "python" + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = "sphinx_rtd_theme" +html_static_path = ["_static"] diff --git a/source/debian-packages.rst b/source/debian-packages.rst new file mode 100644 index 000000000..5306f2882 --- /dev/null +++ b/source/debian-packages.rst @@ -0,0 +1,59 @@ +Package Preparation +================== + +Debian Packages +--------------- + +Introduction to Debian packaging +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Debian packaging provides a standardized way to create software packages for Debian-based operating systems. It ensures easy installation, upgrade, and removal of software components. The packaging format used in Debian is known as the Debian package format or `.deb`. + +Debian-specific tools and conventions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Debian packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following: + +- **dpkg**: The Debian package manager (`dpkg`) is the core tool for installing, querying, and managing Debian packages. It operates at a lower level than higher-level package management tools like `apt` or `apt-get`. + +- **Debian control file**: The `debian/control` file is a metadata file that provides information about the package, its dependencies, and other package-related details. It includes fields such as package name, version, maintainer, description, dependencies, and more. Maintaining an accurate and up-to-date control file is essential for proper package management. + +- **Debian changelog**: The `debian/changelog` file maintains a record of changes made to the package across different versions. It includes information about the package version, the person responsible for the change, the change description, and other relevant details. The changelog helps users and maintainers track the history of the package and understand the modifications introduced in each release. + +- **Debian rules file**: The `debian/rules` file is a makefile-like script that defines the build process and other package-specific tasks. It specifies how to compile the source code, configure the package, and install the files in the correct locations. The rules file allows for customization and automation of the packaging process. + +Other important Debian-specific tools and files include `debuild`, `lintian`, and `pbuilder`, which aid in building, linting, and testing packages respectively. + +Additional Debian Files +~~~~~~~~~~~~~~~~~~~~~~ + +In addition to the core Debian packaging files, the packaging process may involve other files with specific purposes. Here are some of the additional Debian files commonly used: + +- **Source/format**: This file specifies the format version of the Debian source package. + +- **Source/lintian-overrides**: This file is used to override specific lintian warnings or errors generated during the package build process. + +- **Tests/control**: This file contains information about the tests to be run on the package. + +- **Tests/installcheck**: This file specifies commands or scripts to be executed after the package is installed. + +- **Upstream/signing-key.asc**: This file contains the ASCII-armored representation of the upstream signing key used to verify the authenticity and integrity of the package. + +- **Changelog**: The `debian/changelog` file maintains a record of changes made to the package. + +- **Compat**: The `debian/compat` file specifies the compatibility level of the package's packaging format. + +- **Control.in**: The `debian/control.in` file is a template file used to generate the `debian/control` file. + +- **Copyright**: The `debian/copyright` file provides information about the copyright holders and licensing terms. + +- **Docs**: The `debian/docs` file lists the additional documentation files included with the package. + +- **NOTICE**: The `debian/NOTICE` file contains any legally required notices or attributions. + +- **Pgversion**: The `debian/pgversion` file specifies the PostgreSQL version for which the package is intended. + +- **Rules**: The `debian/rules` file is a makefile-like script that defines the build process. + + +For more information about Debian packaging, see the [Debian New Maintainers' Guide](https://www.debian.org/doc/manuals/maint-guide/). \ No newline at end of file diff --git a/source/index.rst b/source/index.rst new file mode 100644 index 000000000..ae0aea95c --- /dev/null +++ b/source/index.rst @@ -0,0 +1,40 @@ +.. Django Multi-tenant documentation master file, created by + sphinx-quickstart on Mon Feb 13 13:32:28 2023. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to ActiveRecord Multi-tenant's documentation! +====================================================== + +|Build Status| |Latest Documentation Status| + +.. |Build Status| image:: https://github.com/citusdata/activerecord-multi-tenant/actions/workflows/active-record-multi-tenant-tests.yml/badge.svg + :target: https://github.com/citusdata/activerecord-multi-tenant/actions/workflows/active-record-multi-tenant-tests.yml + :alt: Build Status + + +.. |Latest Documentation Status| image:: https://readthedocs.org/projects/django-multitenant/badge/?version=latest + :target: https://activerecord-multi-tenant.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status + + +Citus Data Packaging Documentation +======================================== + +Welcome to the official documentation for ``CitusData Packaging``. + +.. toctree:: + :maxdepth: 2 + :caption: Table of Contents + + introduction + package-build-environment + repository-installation-scripts + debian-packages + rpm-packages + tool-scripts + project-packaging-configurations + citus-docker-images + pgdg + pgxn + diff --git a/source/introduction.rst b/source/introduction.rst new file mode 100644 index 000000000..e11cb9dc6 --- /dev/null +++ b/source/introduction.rst @@ -0,0 +1,47 @@ +Packaging Process Documentation +============================== + +Introduction +------------ + +Welcome to the documentation outlining the packaging process for Citus and its associated projects. Packaging plays a crucial role in the distribution and deployment of software, ensuring that it can be easily installed and utilized across various operating systems and environments. + +In this document, we will explore the steps and best practices involved in preparing packages for different operating systems and releases. We will focus on the Debian and RPM package formats, as they are widely used and supported in the industry. Additionally, we will cover the creation and management of Docker environments to facilitate packaging for specific OS/release/PostgreSQL version combinations. + +The supported operating systems and releases for packaging include: + +- CentOS 8 +- CentOS 7 +- Oracle Linux 8 +- Oracle Linux 7 +- AlmaLinux 9 +- Debian Buster +- Debian Bullseye +- Debian Bookworm +- Ubuntu Bionic +- Ubuntu Focal +- Ubuntu Jammy +- Ubuntu Kinetic + +The packaging process will involve preparing packages for various projects, including: + +- `Citus `_ +- `Pgazure `_ +- Cron + - `Debian `_ + - `RPM `_ +- Hll + - `Debian `_ + - `RPM `_ +- Topn + - `Debian `_ + - `RPM `_ +- `Azure-gdpr `_ + +Once the packages are ready, we will push them to PackageCloud, a package management and distribution service. Additionally, we will create Docker images for each OS/release/PostgreSQL version combination, which will be published on Docker Hub. + +Lastly, we will cover the publishing of the applications into PGXN (PostgreSQL Extension Network), a platform that hosts and distributes PostgreSQL extensions. + +By following the steps outlined in this document, you will gain a comprehensive understanding of the packaging process for Citus and related projects. This knowledge will empower you to efficiently distribute and deploy the software across a wide range of operating systems and environments. + +Now, let's dive into the detailed steps for package preparation, Docker environment management, Docker image creation, PackageCloud integration, and PGXN publishing. diff --git a/source/package-build-environment.rst b/source/package-build-environment.rst new file mode 100644 index 000000000..390958459 --- /dev/null +++ b/source/package-build-environment.rst @@ -0,0 +1,97 @@ +III. Packaging Build Environment Management +--------------------------------- + +A. Introduction to Packaging environments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Mainly, we are using docker images for each OS/release/postgres version pair. We are using docker images to create a reproducible and consistent packaging environment. Docker images provide a reliable and consistent way to package software applications. They offer several advantages for packaging processes: +1. Isolation: Docker containers encapsulate applications and their dependencies, ensuring isolation from the underlying host system. This isolation helps prevent conflicts between different software components and provides a clean and reproducible environment for packaging. +2. Reproducibility: Docker environments enable the creation of reproducible packaging environments. By defining the dependencies and configurations within a Docker image, it becomes easier to recreate the same environment across different systems, ensuring consistent packaging results. +3. Portability: Docker environments are portable and can be easily shared across different systems. This portability ensures that the same packaging results can be achieved across different systems, eliminating environment-related issues. +4. Scalability: Docker environments can be easily scaled up or down to meet the packaging requirements. This scalability ensures that the packaging process can be easily scaled up or down to meet the packaging requirements. +5. Other: Docker environments provide a secure,reliable,consistent,flexible,efficient,simple and maintainable packaging environment. Docker environments provide this by isolating the packaging process from the underlying host system. + +B. Docker images for various package types +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We are using docker images for each OS/release/postgres version pair. We are using docker images to create a reproducible and consistent packaging environment. +For debian based packages, we can use the same docker image for all postgres versions. For example, we can use the same docker image for debian/10 postgres 13 and debian/10 postgres 14 packages. +For rpm based packages, we need a dedicated docker image for each postgres version. For example, we need a docker image for centos/7 postgres 13 and centos/7 postgres 14 packages. + +Below is our supported os/release pair + +1. Centos + 1. centos/7 +2. Oracle Linux + 1. oraclelinux/7 + 2. oraclelinux/8 +3. AlmaLinux + 1. almalinux/9 +4. Debian + 1. debian/buster + 2. debian/bullseye + 3. debian/bookworm +5. Ubuntu + 1. ubuntu/bionic + 2. ubuntu/focal + 3. ubuntu/jammy + 4. ubuntu/kinetic +6. pgxn + +Below is our supported postgres version +1. 13 +2. 14 +3. 15 +4. 16 (planned) + +Therefore, docker images for each OS/release/postgres version pair are as follows: + +1. Centos + 1. centos/7 postgres 13 + 2. centos/7 postgres 14 + 3. centos/7 postgres 15 + 4. centos/7 postgres 16 (planned) +2. Oracle Linux + 1. oraclelinux/7 postgres 13 + 2. oraclelinux/7 postgres 14 + 3. oraclelinux/7 postgres 15 + 4. oraclelinux/7 postgres 16 (planned) + 5. oraclelinux/8 postgres 13 + 6. oraclelinux/8 postgres 14 + 7. oraclelinux/8 postgres 15 + 8. oraclelinux/8 postgres 16 (planned) +3. AlmaLinux + 1. almalinux/9 postgres 13 + 2. almalinux/9 postgres 14 + 3. almalinux/9 postgres 15 + 4. almalinux/9 postgres 16 (planned) +4. Debian + 1. debian/buster all postgres versions + 2. debian/bullseye all postgres versions + 3. debian/bookworm all postgres versions +5. Ubuntu + 1. ubuntu/bionic all postgres versions + 2. ubuntu/focal all postgres versions + 3. ubuntu/jammy all postgres versions + 4. ubuntu/kinetic all postgres versions +6. Pgxn + +C. Developing and Maintaining Docker Images +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Dockerfiles, which are used to build docker images, are located in the develop branch of packaging repository. +We have a template structure to auto-generate the docker files for each OS/release/postgres version pair. The template structure is as follows: + +1. `Dockerfile-deb.tmpl `_: This template is used to generate docker files for debian based docker files. +2. `Dockerfile-rpm.tmpl `_ : This template is used to generate docker files for rpm based docker files. +3. `Dockerfile-pgxn.tmpl `_ : This template is used to generate docker files for pgxn based docker files. + +After changing the template files, we need to run the `update_dockerfiles.sh `_ script to generate the docker files for each OS/release/postgres version pair. +After commit and push the changes, GH actions will build the docker images using the generated docker files. After seeing all the docker images are built successfully, we can merge the changes to the master branch. +When we merge the changes to the master branch, GH actions will push the docker images to the `citus/packaging `_ docker hub repository. + +If you want to publish test images from `citus/packaging-test `_, you can use the `test pipeline `_. +To use the test pipeline, you need to change the current branch into your branch name. In this case, the test pipeline will push the docker images to the `citus/packaging-test `_ docker hub repository. + + + diff --git a/source/pgdg.rst b/source/pgdg.rst new file mode 100644 index 000000000..7cb77fd7a --- /dev/null +++ b/source/pgdg.rst @@ -0,0 +1,15 @@ +Citus PGDG Packages +==================== + +PGDG is a repository of PostgreSQL packages for several Linux distributions. +Citus packages are available for RPM based distributions. +When releasing Citus, we open an issue in Postgres Redmine to request a new release from the link below +https://redmine.postgresql.org/projects/pgrpms/issues/new +We select the fields as below: +* Tracker: Bug +* Subject: https://redmine.postgresql.org/projects/pgrpms/issues/new +* Description: Kindly release the following version of Citus: xx.x. +* Category: New Package +* Priority: Normal +* Target version: +* Assignee: Devrim Gündüz diff --git a/source/pgxn.rst b/source/pgxn.rst new file mode 100644 index 000000000..1f79e3134 --- /dev/null +++ b/source/pgxn.rst @@ -0,0 +1,35 @@ +Citus PGXN Extension Builds +=========================== + +PGXN Build Infrastructure +---------------------------- +In packaging system, we have scripts and images to build PGXN extensions. +In package build environment, we have Docker images , so Dockerfiles, to build pgxn extensions. +`Docker image `_ +is the base image for all pgxn extensions. It has all the dependencies to build pgxn extensions. +In this docker image, there is a script called `fetch_and_build_pgxn `_ +which is used to build pgxn extensions. + +Citus PGXN COnfiguration +------------------------ + +As each project package configuration, we have a branch named `pgxn-citus `_ +which is used to build pgxn extension for citus. In this branch, we have three configuration files used to build the packages +1. `META.json _ +2. pkgvars +3. postgres-matrix.yml + +Pkgvars and postgres-matrix.yml has the same usage as in other project builds. + +Building PGXN Extension for the new release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +When we build the pgxn extenson for citus, we need to go through the following steps. +1. Execute the pipeline `Update Version on PGXN Config Files `_ + to update the version in the pgxn configuration files. This pipeline will create a PR with the updated version. +2. Check for the minimum postgres version. If it is different from the one that citus has, it should be changed to the one that citus has. +3. Check for the tests and if they all pass, notify the PR reviewer to review the PR. +4. Once the PR is reviewed, merge the PR and make sure that the `Citus PGXN `_ is updated with the new version. + + + + diff --git a/source/project-packaging-configurations.rst b/source/project-packaging-configurations.rst new file mode 100644 index 000000000..f9c06eb03 --- /dev/null +++ b/source/project-packaging-configurations.rst @@ -0,0 +1,69 @@ +Project Packaging Configurations +================================= + +For each project, we open a new branch in the packaging repository. +In these branches, we add the packaging configuration files for the project. The packaging configuration files are: +1. Pkgvars +2. Postgres-matrix.yml + +Pkgvars +------- +Pkgvars is a property file that contains the basic information about the package. It is used by the packaging system to generate the package name, version, etc. + +Below is a sample pkgvars file: + +``` +pkgname=citus +hubproj=citus +pkgdesc='Citus (Open-Source)' +pkglatest=11.3.0.citus-1 +nightlyref=main +versioning=fancy + +``` + +Parameters for the pkgvars file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1. pkgname: prefix of the package +2. hubproj: name of the repository in the GitHub +3. pkgdesc: description of the package. It is just for reference. It is not used by the packaging system +4. pkglatest: latest version of the package. It is used by the packaging system to generate the package name and get the release number +from the GitHub repository +5. nightlyref: branch name of the nightly builds +6. versioning: versioning scheme. It can be either `fancy` or `simple`. If it is `fancy`,fancy version numbers are added to the package name. +By default, it is `simple`. + +Postgres-matrix.yml +------- +Citus Postgres version matrix. Packaging system decides which postgres versions for the Citus version by referencing this file + +Below is a sample postgres-matrix.yml file: + +``` +name: Postgres Version Matrix +project_name: citus + +version_matrix: + - 8.0: + postgres_versions: [10, 11] + - 9.0: + postgres_versions: [11, 12] + - 9.5: + postgres_versions: [11, 12, 13] + - 10.1: + postgres_versions: [12, 13] + - 10.2: + postgres_versions: [12, 13, 14] + - 11.0: + postgres_versions: [13, 14] + - 11.1: + postgres_versions: [ 13, 14, 15 ] + +``` + +In packaging system, version number is taken from pkgvars file and correct matrix entry is being found from this file by comparing the version number with the matrix entries. +Then, the postgres versions are taken from the matrix entry and one package is generated for each postgres version. + +For example, if we want to create a package for 9.3.8 version of Citus, we will take the matrix entry for 9.0 and +create a package for each postgres version in the matrix entry since 9.3.8 is between 9.0 and 9.5. + diff --git a/source/repository-installation-scripts.rst b/source/repository-installation-scripts.rst new file mode 100644 index 000000000..554725591 --- /dev/null +++ b/source/repository-installation-scripts.rst @@ -0,0 +1,47 @@ +Repository Installation Scripts +=============================== + +Citus packages are stored in packagecloud.io. To install the packages, you can use the packagecloud.io installation scripts directly. +However, custom installation scripts are provided for convenience. Benefits of using the custom installation scripts are: +1. install.citusdata.com/repos.citusdata.com is used as the repository URL instead of packagecloud.io. These URLs are our own URLs + and isolating users from packagecloud.io which is useful in case we need to change the packagecloud.io URL in the future. +2. Since our packages directly related to postgresql, we install postgresql repository for convenience. +3. Programs like curl, wget, gnupg, apt-transport-https are installed if they are not already installed which are required for installing postgres and citus packages. + +Repository Installation Scripts are stored in the `packaging/gh-pages branch `_. +Files stored in this branch are served using GH pages. +You can see the GH Pages settings in the `settings `_. +The original url of the files are `https://citusdata.github.io/packaging/`_. +For example, the url of the `deb.sh < https://citusdata.github.io/packaging/community/deb.sh>`_. +install.citusdata.com/repos.citusdata.com is a CNAME record for citusdata.github.io in our DNS settings in Cloudflare. +For more information about url redirection, please refer to Citus Packaging Web Url Certificates document + +There are 8 scripts in this branch for 4 repositories the packagecloud.io. +1. community + * `deb.sh `_ + * `rpm.sh `_ +2. community-nightlies + * `deb.sh `_ + * `rpm.sh `_ +3. enterprise + * `deb.sh `_ + * `rpm.sh `_ +4. enterprise-nightlies + * `deb.sh `_ + * `rpm.sh `_ + +These scripts are used to install the repositories. For example, to install the community repository, you can run the following command: + +For Debian/Ubuntu for community repository: + +``` +curl https://install.citusdata.com/community/deb.sh | sudo bash +``` + +For RHEL/CentOS for community repository: + +``` +curl https://install.citusdata.com/community/rpm.sh | sudo bash +``` + + diff --git a/source/rpm-packages.rst b/source/rpm-packages.rst new file mode 100644 index 000000000..47a1bb041 --- /dev/null +++ b/source/rpm-packages.rst @@ -0,0 +1,30 @@ +RPM Packages +------------ + +Introduction to RPM packaging +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +RPM packaging is a common method used to create software packages for RPM-based Linux distributions. RPM (Red Hat Package Manager) provides a reliable and consistent way to install, upgrade, and remove software components. The packaging format used in RPM is known as the RPM package format or `.rpm`. + +RPM-specific tools and conventions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +RPM packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following: + +- **rpmbuild**: The `rpmbuild` command is the primary tool for building RPM packages. It compiles the source code, creates the package structure, and generates the final RPM package. + +- **.spec file**: The `.spec` file is a key component of RPM packaging. It contains detailed instructions and metadata about the package, such as its name, version, release, dependencies, build requirements, and file locations. The `.spec` file defines the build process, file ownership, permissions, and other package-specific configurations. + +- **rpmlintrc**: The `rpmlintrc` file is used to configure the rpmlint tool, which performs static analysis on the generated RPM package. The rpmlint tool checks for common packaging issues, compliance with guidelines, and potential errors. The `rpmlintrc` file allows you to specify specific checks to enable or disable for your package. + +Additional RPM Files +~~~~~~~~~~~~~~~~~~~~ + +In addition to the core RPM packaging components, the packaging process may involve other files with specific purposes. Here are some additional RPM-related files commonly used: + +- **.spec file**: The `.spec` file contains the build instructions and metadata for the RPM package. + +- **rpmlintrc**: The `rpmlintrc` file configures the rpmlint tool for checking the package. + +For more information about RPM packaging, see the [RPM Packaging Guide](https://rpm-packaging-guide.github.io/). + diff --git a/source/tools-scripts.rst b/source/tools-scripts.rst new file mode 100644 index 000000000..75dfe28eb --- /dev/null +++ b/source/tools-scripts.rst @@ -0,0 +1,10 @@ +Packaging Tools Scripts +======================= + +Packaging operations are performed by a set of python scripts stored under the `tools `_ repository. +The scripts are written in python and require python 3.8 or higher. The scripts are tested on Ubuntu focal. + +For each change we add a tag to the commit and push it to the repository. The tag is of the form `v..` format. +Mainly, we are using the scripts inside GH Actions pipelines. In these pipelines we use the tools scripts with the corresponding tag. + +For more details on the tools scripts, please refer to the `tools README `_. \ No newline at end of file From ab4abe8df20c53c78e8c9d61b1fda01a0860532d Mon Sep 17 00:00:00 2001 From: gindibay Date: Fri, 30 Jun 2023 13:29:00 +0300 Subject: [PATCH 02/11] Updates docs content --- source/conf.py | 1 + source/debian-packages.rst | 8 +- source/index.rst | 43 ++++-- source/introduction.rst | 29 ++-- source/package-build-environment.rst | 160 +++++++++++--------- source/packaging-process.rst | 150 ++++++++++++++++++ source/pgdg.rst | 11 +- source/pgxn.rst | 17 +-- source/project-packaging-configurations.rst | 49 +++--- source/repository-installation-scripts.rst | 67 ++++---- 10 files changed, 354 insertions(+), 181 deletions(-) create mode 100644 source/packaging-process.rst diff --git a/source/conf.py b/source/conf.py index a526b2e94..b615f09c9 100644 --- a/source/conf.py +++ b/source/conf.py @@ -28,3 +28,4 @@ html_theme = "sphinx_rtd_theme" html_static_path = ["_static"] + diff --git a/source/debian-packages.rst b/source/debian-packages.rst index 5306f2882..3d975ba1a 100644 --- a/source/debian-packages.rst +++ b/source/debian-packages.rst @@ -1,11 +1,11 @@ Package Preparation -================== +==================== Debian Packages ---------------- +---------------- Introduction to Debian packaging -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Debian packaging provides a standardized way to create software packages for Debian-based operating systems. It ensures easy installation, upgrade, and removal of software components. The packaging format used in Debian is known as the Debian package format or `.deb`. @@ -25,7 +25,7 @@ Debian packaging relies on a set of tools and conventions to streamline the pack Other important Debian-specific tools and files include `debuild`, `lintian`, and `pbuilder`, which aid in building, linting, and testing packages respectively. Additional Debian Files -~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~ In addition to the core Debian packaging files, the packaging process may involve other files with specific purposes. Here are some of the additional Debian files commonly used: diff --git a/source/index.rst b/source/index.rst index ae0aea95c..bd9566144 100644 --- a/source/index.rst +++ b/source/index.rst @@ -1,9 +1,5 @@ -.. Django Multi-tenant documentation master file, created by - sphinx-quickstart on Mon Feb 13 13:32:28 2023. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. -Welcome to ActiveRecord Multi-tenant's documentation! +Welcome to Citus Data documentation! ====================================================== |Build Status| |Latest Documentation Status| @@ -23,18 +19,47 @@ Citus Data Packaging Documentation Welcome to the official documentation for ``CitusData Packaging``. +``CitusData Packaging`` is a collection of scripts and tools that are used to packaging Citus Data products. +This documentation is intended to provide a detailed overview of the packaging process and the tools used to package Citus Data products. + + + +.Table of Contents +=================== + +Introduction +------------ .. toctree:: - :maxdepth: 2 - :caption: Table of Contents + :maxdepth: 1 introduction package-build-environment repository-installation-scripts + +Tools and Packages +------------------ +.. toctree:: + :maxdepth: 1 + + tools-scripts debian-packages rpm-packages - tool-scripts - project-packaging-configurations + +Citus Related +------------- +.. toctree:: + :maxdepth: 1 + citus-docker-images pgdg pgxn +Project Packaging +----------------- +.. toctree:: + :maxdepth: 1 + + project-packaging-configurations + packaging-process + + diff --git a/source/introduction.rst b/source/introduction.rst index e11cb9dc6..7444406dd 100644 --- a/source/introduction.rst +++ b/source/introduction.rst @@ -1,8 +1,5 @@ -Packaging Process Documentation -============================== - Introduction ------------- +=============================== Welcome to the documentation outlining the packaging process for Citus and its associated projects. Packaging plays a crucial role in the distribution and deployment of software, ensuring that it can be easily installed and utilized across various operating systems and environments. @@ -25,18 +22,18 @@ The supported operating systems and releases for packaging include: The packaging process will involve preparing packages for various projects, including: -- `Citus `_ -- `Pgazure `_ -- Cron - - `Debian `_ - - `RPM `_ -- Hll - - `Debian `_ - - `RPM `_ -- Topn - - `Debian `_ - - `RPM `_ -- `Azure-gdpr `_ +* `Citus `_ +* `Pgazure `_ +* Cron + * `Cron Debian `_ + * `Cron RPM `_ +* Hll + * `Hll Debian `_ + * `Hll RPM `_ +* Topn + * `Topn Debian `_ + * `Topn RPM `_ +* `Azure-gdpr `_ Once the packages are ready, we will push them to PackageCloud, a package management and distribution service. Additionally, we will create Docker images for each OS/release/PostgreSQL version combination, which will be published on Docker Hub. diff --git a/source/package-build-environment.rst b/source/package-build-environment.rst index 390958459..9176e86b5 100644 --- a/source/package-build-environment.rst +++ b/source/package-build-environment.rst @@ -1,97 +1,107 @@ -III. Packaging Build Environment Management ---------------------------------- +Packaging Build Environment Management +====================================== -A. Introduction to Packaging environments -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduction to Packaging Environments +-------------------------------------- -Mainly, we are using docker images for each OS/release/postgres version pair. We are using docker images to create a reproducible and consistent packaging environment. Docker images provide a reliable and consistent way to package software applications. They offer several advantages for packaging processes: -1. Isolation: Docker containers encapsulate applications and their dependencies, ensuring isolation from the underlying host system. This isolation helps prevent conflicts between different software components and provides a clean and reproducible environment for packaging. -2. Reproducibility: Docker environments enable the creation of reproducible packaging environments. By defining the dependencies and configurations within a Docker image, it becomes easier to recreate the same environment across different systems, ensuring consistent packaging results. -3. Portability: Docker environments are portable and can be easily shared across different systems. This portability ensures that the same packaging results can be achieved across different systems, eliminating environment-related issues. -4. Scalability: Docker environments can be easily scaled up or down to meet the packaging requirements. This scalability ensures that the packaging process can be easily scaled up or down to meet the packaging requirements. -5. Other: Docker environments provide a secure,reliable,consistent,flexible,efficient,simple and maintainable packaging environment. Docker environments provide this by isolating the packaging process from the underlying host system. +In our packaging process, we utilize Docker images for each OS/release/PostgreSQL version combination. Docker images offer a reliable and consistent environment for packaging software applications. They provide several advantages for the packaging process: -B. Docker images for various package types -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1. **Isolation**: Docker containers encapsulate applications and their dependencies, ensuring isolation from the underlying host system. This isolation helps prevent conflicts between different software components and provides a clean and reproducible environment for packaging. -We are using docker images for each OS/release/postgres version pair. We are using docker images to create a reproducible and consistent packaging environment. -For debian based packages, we can use the same docker image for all postgres versions. For example, we can use the same docker image for debian/10 postgres 13 and debian/10 postgres 14 packages. -For rpm based packages, we need a dedicated docker image for each postgres version. For example, we need a docker image for centos/7 postgres 13 and centos/7 postgres 14 packages. +2. **Reproducibility**: Docker environments enable the creation of reproducible packaging environments. By defining the dependencies and configurations within a Docker image, it becomes easier to recreate the same environment across different systems, ensuring consistent packaging results. -Below is our supported os/release pair +3. **Portability**: Docker environments are portable and can be easily shared across different systems. This portability ensures that the same packaging results can be achieved across different systems, eliminating environment-related issues. + +4. **Scalability**: Docker environments can be easily scaled up or down to meet the packaging requirements. This scalability ensures that the packaging process can be efficiently adjusted to handle different workload sizes. + +5. **Other Benefits**: Docker environments provide a secure, reliable, consistent, flexible, efficient, simple, and maintainable packaging environment. Docker isolates the packaging process from the underlying host system, ensuring a controlled and stable environment. + +Docker Images for Various Package Types +--------------------------------------- + +We utilize Docker images for each OS/release/PostgreSQL version combination to create a reproducible and consistent packaging environment. + +For Debian-based packages, we can use the same Docker image for all PostgreSQL versions. + +For example, the Debian/10 Docker image can be used for both PostgreSQL 13 and PostgreSQL 14 packages. + +However, for RPM-based packages, we need a dedicated Docker image for each PostgreSQL version. + +For instance, we require a Docker image for CentOS/7 with PostgreSQL 13 and another Docker image for CentOS/7 with PostgreSQL 14 packages. + +Here are the supported OS/release pairs: + +1. CentOS + - centos/7 -1. Centos - 1. centos/7 2. Oracle Linux - 1. oraclelinux/7 - 2. oraclelinux/8 + - oraclelinux/7 + - oraclelinux/8 + 3. AlmaLinux - 1. almalinux/9 + - almalinux/9 + 4. Debian - 1. debian/buster - 2. debian/bullseye - 3. debian/bookworm + - debian/buster + - debian/bullseye + - debian/bookworm + 5. Ubuntu - 1. ubuntu/bionic - 2. ubuntu/focal - 3. ubuntu/jammy - 4. ubuntu/kinetic -6. pgxn - -Below is our supported postgres version -1. 13 -2. 14 -3. 15 -4. 16 (planned) - -Therefore, docker images for each OS/release/postgres version pair are as follows: - -1. Centos - 1. centos/7 postgres 13 - 2. centos/7 postgres 14 - 3. centos/7 postgres 15 - 4. centos/7 postgres 16 (planned) + - ubuntu/bionic + - ubuntu/focal + - ubuntu/jammy + - ubuntu/kinetic + +6. PGXN + +Based on these OS/release pairs, the Docker images for each OS/release/PostgreSQL version combination are as follows: + +1. CentOS + - centos/7 PostgreSQL 13 + - centos/7 PostgreSQL 14 + - centos/7 PostgreSQL 15 + - centos/7 PostgreSQL 16 (planned) 2. Oracle Linux - 1. oraclelinux/7 postgres 13 - 2. oraclelinux/7 postgres 14 - 3. oraclelinux/7 postgres 15 - 4. oraclelinux/7 postgres 16 (planned) - 5. oraclelinux/8 postgres 13 - 6. oraclelinux/8 postgres 14 - 7. oraclelinux/8 postgres 15 - 8. oraclelinux/8 postgres 16 (planned) + - oraclelinux/7 PostgreSQL 13 + - oraclelinux/7 PostgreSQL 14 + - oraclelinux/7 PostgreSQL 15 + - oraclelinux/7 PostgreSQL 16 (planned) + - oraclelinux/8 PostgreSQL 13 + - oraclelinux/8 PostgreSQL 14 + - oraclelinux/8 PostgreSQL 15 + - oraclelinux/8 PostgreSQL 16 (planned) 3. AlmaLinux - 1. almalinux/9 postgres 13 - 2. almalinux/9 postgres 14 - 3. almalinux/9 postgres 15 - 4. almalinux/9 postgres 16 (planned) + - almalinux/9 PostgreSQL 13 + - almalinux/9 PostgreSQL 14 + - almalinux/9 PostgreSQL 15 + - almalinux/9 PostgreSQL 16 (planned) 4. Debian - 1. debian/buster all postgres versions - 2. debian/bullseye all postgres versions - 3. debian/bookworm all postgres versions + - debian/buster (all PostgreSQL versions) + - debian/bullseye (all PostgreSQL versions) + - debian/bookworm (all PostgreSQL versions) 5. Ubuntu - 1. ubuntu/bionic all postgres versions - 2. ubuntu/focal all postgres versions - 3. ubuntu/jammy all postgres versions - 4. ubuntu/kinetic all postgres versions -6. Pgxn + - ubuntu/bionic (all PostgreSQL versions) + - ubuntu/focal (all PostgreSQL versions) + - ubuntu/jammy (all PostgreSQL versions) + - ubuntu/kinetic (all PostgreSQL versions) +6. PGXN + +Developing and Maintaining Docker Images +----------------------------------------- -C. Developing and Maintaining Docker Images -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The Dockerfiles used to build Docker images are located in the "develop" branch of the packaging repository. We follow a template structure to auto-generate Docker files for each OS/release/PostgreSQL version combination. The templates used are: -Dockerfiles, which are used to build docker images, are located in the develop branch of packaging repository. -We have a template structure to auto-generate the docker files for each OS/release/postgres version pair. The template structure is as follows: +1. `Dockerfile-deb.tmpl `_: This template generates Docker files for Debian-based packages. +2. `Dockerfile-rpm.tmpl `_: This template generates Docker files for RPM-based packages. +3. `Dockerfile-pgxn.tmpl `_: This template generates Docker files for PGXN-based packages. -1. `Dockerfile-deb.tmpl `_: This template is used to generate docker files for debian based docker files. -2. `Dockerfile-rpm.tmpl `_ : This template is used to generate docker files for rpm based docker files. -3. `Dockerfile-pgxn.tmpl `_ : This template is used to generate docker files for pgxn based docker files. +After making changes to the template files, the `update_dockerfiles.sh `_ script needs to be run to generate the Docker files for each OS/release/PostgreSQL version combination. -After changing the template files, we need to run the `update_dockerfiles.sh `_ script to generate the docker files for each OS/release/postgres version pair. -After commit and push the changes, GH actions will build the docker images using the generated docker files. After seeing all the docker images are built successfully, we can merge the changes to the master branch. -When we merge the changes to the master branch, GH actions will push the docker images to the `citus/packaging `_ docker hub repository. +Once the changes are committed and pushed, the GitHub Actions workflow will build the Docker images using the generated Docker files. -If you want to publish test images from `citus/packaging-test `_, you can use the `test pipeline `_. -To use the test pipeline, you need to change the current branch into your branch name. In this case, the test pipeline will push the docker images to the `citus/packaging-test `_ docker hub repository. +After confirming that all Docker images are successfully built, the changes can be merged into the master branch. +When the changes are merged into the master branch, the GitHub Actions workflow will push the Docker images to the `citus/packaging `_ Docker Hub repository. +If you want to publish test images from `citus/packaging-test `_, you can use the `test pipeline `_. To use the test pipeline, change the current branch to your branch name. The test pipeline will then push the Docker images to the `citus/packaging-test `_ Docker Hub repository. diff --git a/source/packaging-process.rst b/source/packaging-process.rst new file mode 100644 index 000000000..29dad2fc7 --- /dev/null +++ b/source/packaging-process.rst @@ -0,0 +1,150 @@ +Packaging Process +================= + +Packaging process is done by the following steps: + +1. Bake deb and rpm packages +2. Bake docker images +3. Bake pgxn package +4. Open issue for PGDG packaging + +Bake deb and rpm packages +--------------------------- + +In this phase, we need to edit packaging configuration files to bake the packages for the desired application version. + +Editing Configuration Files by Pipeline +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Citus has its own automation to edit the packaging configuration scripts; `Update Package Properties `_ workflow. +After executing this workflow on all-citus branch, you will see a new PR to the packaging repository. This PR will update the packaging configuration files to the desired version. +After seeing all the checks are passed, you can request a review from the packaging team. +After the review is done, you can merge the PR. +After all jobs are done, you can see the new packages in the `Citusdata Package Repository `_. +You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. + +Editing Configuration Files Manually +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Other projects such as pg_azure_storage, topn does not have automation pipelines, so we need to edit the configuration files manually. +There are two categories of projects that we need to edit the configuration files manually. + +1. Projects that deb and rpm configuration files are in the same branch +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +``pg_azure_storage``, ``azure-gdpr`` is in this category. +Below are the steps to edit the configuration files manually: + +1. Create a new branch from the latest commit on the master branch. +2. Edit the configuration files to the desired version. + * Add a new entry to the ``debian/changelog`` file. + * Update the version in the pkgvars file + * Update the version in the rpm spec file + * Add a new changelog entry to the rpm spec file +3. Create a PR from the new branch to the master branch. +4. After the PR is merged, you can see the new packages in the `Citusdata Package Repository `_. You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. + +2. Projects that deb and rpm configuration files are in different branches +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +``topn``, ``hll`` and ``cron`` is in this category. +Below are the steps to edit the configuration files manually: + +Debian Configuration Files +''''''''''''''''''''''''''' + +1. Checkout the debian- branch. +2. Create a new branch from the latest commit on the debian- branch. +3. Edit the configuration files to the desired version. + * Add a new entry to the ``debian/changelog`` file. + * Update the version in the pkgvars file +4. Create a PR from the new branch to the debian- branch. +5. After the PR is merged, you can see the new packages in the `Citusdata Package Repository `_. You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. + +Redhat Configuration Files +''''''''''''''''''''''''''' + +1. Checkout the rpm- branch. +2. Create a new branch from the latest commit on the rpm- branch. +3. Edit the configuration files to the desired version. + * Update the version in the rpm spec file + * Add a new changelog entry to the rpm spec file +4. Create a PR from the new branch to the rpm- branch. +5. After the PR is merged, you can see the new packages in the `Citusdata Package Repository `_. You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. + +Bake docker images +------------------ +This step is applicable just for Citus Community. + +Baking Main versions +~~~~~~~~~~~~~~~~~~~~ +In this phase, we need to edit the docker image configuration files to bake the docker images for the desired application version. +To edit the docker image configuration files, you can use the `Update Version on Docker Files `_ workflow. +After executing this workflow on master branch, you will see a new PR to the docker repository. This PR will update the docker image configuration files to the desired version. +After seeing all the checks are passed, you can request a review from the packaging team. +After the review is done, you can merge the PR. +After all jobs are done, you can see the new docker images in the `Citusdata Docker Repository `_. +Versions that needs to be in the docker repository are: +1. latest +2. alpine +3. postgres_xx (all supported postgres versions before latest) + +Create Images for Patch Releases +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In this phase, we need to create docker images for patch releases. For example, if we have a patch release for 10.1.1, we need to create a docker image for 10, 10.1 and 10.1.1. + +To create docker images for patch releases, you just need to add a tag to the latest commit on master branch and push it +`The tag pipeline `_ will create the docker images for the tag and push them to the docker repository. +After all jobs are done, you can see the new docker images in the `Citusdata Docker Repository `_. +The versioned images should appear in the docker repository for all supported postgres versions and alpine. +For example for 10.1.1 for postgres 13,14,15; we need to see the following images in the docker repository: + +1. 10.1.1 (latest i.e. 15) +2. 10.1 (latest i.e. 15) +3. 10 (latest i.e. 15) +4. 10.1.1-alpine +5. 10.1-alpine +6. 10-alpine +7. 10.1.1-pg13 +8. 10.1-pg13 +9. 10-pg13 +10. 10.1.1-pg14 +11. 10.1-pg14 +12. 10-pg14 + +Bake pgxn packages +-------------------------------------------- +When we build the pgxn extension for citus, we need to go through the following steps. + +1. Execute the pipeline `Update Version on PGXN Config Files `_ to update the version in the pgxn configuration files. This pipeline will create a PR with the updated version. + +2. Check for the minimum postgres version. If it is different from the one that citus has, it should be changed to the one that citus has. + +3. Check for the tests and if they all pass, notify the PR reviewer to review the PR. + +4. Once the PR is reviewed, merge the PR and make sure that the `Citus PGXN `_ is updated with the new version. + +Open issue for PGDG packaging +-------------------------------------------- + +After all the packages are baked, we need to open an issue for PGDG packaging. +PGDG is a repository of PostgreSQL packages for several Linux distributions. +Citus packages are available for RPM based distributions. +When releasing Citus, we open an issue in Postgres Redmine to request a new release from the link below +https://redmine.postgresql.org/projects/pgrpms/issues/new +We select the fields as below: + +* Tracker: Bug + +* Subject: https://redmine.postgresql.org/projects/pgrpms/issues/new + +* Description: Kindly release the following version of Citus: xx.x. + +* Category: New Package + +* Priority: Normal + +* Target version: + +* Assignee: Devrim Gündüz \ No newline at end of file diff --git a/source/pgdg.rst b/source/pgdg.rst index 7cb77fd7a..74e95f579 100644 --- a/source/pgdg.rst +++ b/source/pgdg.rst @@ -3,13 +3,4 @@ Citus PGDG Packages PGDG is a repository of PostgreSQL packages for several Linux distributions. Citus packages are available for RPM based distributions. -When releasing Citus, we open an issue in Postgres Redmine to request a new release from the link below -https://redmine.postgresql.org/projects/pgrpms/issues/new -We select the fields as below: -* Tracker: Bug -* Subject: https://redmine.postgresql.org/projects/pgrpms/issues/new -* Description: Kindly release the following version of Citus: xx.x. -* Category: New Package -* Priority: Normal -* Target version: -* Assignee: Devrim Gündüz +We don't own the process, we open the issue and Devrim Gunduz (PGDG owner) is taking care of it. \ No newline at end of file diff --git a/source/pgxn.rst b/source/pgxn.rst index 1f79e3134..c189fe827 100644 --- a/source/pgxn.rst +++ b/source/pgxn.rst @@ -10,26 +10,19 @@ is the base image for all pgxn extensions. It has all the dependencies to build In this docker image, there is a script called `fetch_and_build_pgxn `_ which is used to build pgxn extensions. -Citus PGXN COnfiguration +Citus PGXN Configuration ------------------------ As each project package configuration, we have a branch named `pgxn-citus `_ which is used to build pgxn extension for citus. In this branch, we have three configuration files used to build the packages + 1. `META.json _ + 2. pkgvars + 3. postgres-matrix.yml Pkgvars and postgres-matrix.yml has the same usage as in other project builds. -Building PGXN Extension for the new release -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When we build the pgxn extenson for citus, we need to go through the following steps. -1. Execute the pipeline `Update Version on PGXN Config Files `_ - to update the version in the pgxn configuration files. This pipeline will create a PR with the updated version. -2. Check for the minimum postgres version. If it is different from the one that citus has, it should be changed to the one that citus has. -3. Check for the tests and if they all pass, notify the PR reviewer to review the PR. -4. Once the PR is reviewed, merge the PR and make sure that the `Citus PGXN `_ is updated with the new version. - - - +.. _Pgxn Build Process: diff --git a/source/project-packaging-configurations.rst b/source/project-packaging-configurations.rst index f9c06eb03..d7c707d13 100644 --- a/source/project-packaging-configurations.rst +++ b/source/project-packaging-configurations.rst @@ -19,7 +19,6 @@ pkgdesc='Citus (Open-Source)' pkglatest=11.3.0.citus-1 nightlyref=main versioning=fancy - ``` Parameters for the pkgvars file @@ -27,39 +26,37 @@ Parameters for the pkgvars file 1. pkgname: prefix of the package 2. hubproj: name of the repository in the GitHub 3. pkgdesc: description of the package. It is just for reference. It is not used by the packaging system -4. pkglatest: latest version of the package. It is used by the packaging system to generate the package name and get the release number -from the GitHub repository +4. pkglatest: latest version of the package. It is used by the packaging system to generate the package name + and get the release number from the GitHub repository 5. nightlyref: branch name of the nightly builds -6. versioning: versioning scheme. It can be either `fancy` or `simple`. If it is `fancy`,fancy version numbers are added to the package name. -By default, it is `simple`. +6. versioning: versioning scheme. It can be either `fancy` or `simple`. If it is `fancy`, + fancy version numbers are added to the package name. By default, it is `simple`. Postgres-matrix.yml -------- +------------------- Citus Postgres version matrix. Packaging system decides which postgres versions for the Citus version by referencing this file Below is a sample postgres-matrix.yml file: -``` -name: Postgres Version Matrix -project_name: citus +.. code-block:: yaml -version_matrix: - - 8.0: - postgres_versions: [10, 11] - - 9.0: - postgres_versions: [11, 12] - - 9.5: - postgres_versions: [11, 12, 13] - - 10.1: - postgres_versions: [12, 13] - - 10.2: - postgres_versions: [12, 13, 14] - - 11.0: - postgres_versions: [13, 14] - - 11.1: - postgres_versions: [ 13, 14, 15 ] - -``` + name: Postgres Version Matrix + project_name: citus + version_matrix: + - 8.0: + postgres_versions: [10, 11] + - 9.0: + postgres_versions: [11, 12] + - 9.5: + postgres_versions: [11, 12, 13] + - 10.1: + postgres_versions: [12, 13] + - 10.2: + postgres_versions: [12, 13, 14] + - 11.0: + postgres_versions: [13, 14] + - 11.1: + postgres_versions: [ 13, 14, 15 ] In packaging system, version number is taken from pkgvars file and correct matrix entry is being found from this file by comparing the version number with the matrix entries. Then, the postgres versions are taken from the matrix entry and one package is generated for each postgres version. diff --git a/source/repository-installation-scripts.rst b/source/repository-installation-scripts.rst index 554725591..cb6f858d7 100644 --- a/source/repository-installation-scripts.rst +++ b/source/repository-installation-scripts.rst @@ -2,46 +2,55 @@ Repository Installation Scripts =============================== Citus packages are stored in packagecloud.io. To install the packages, you can use the packagecloud.io installation scripts directly. -However, custom installation scripts are provided for convenience. Benefits of using the custom installation scripts are: -1. install.citusdata.com/repos.citusdata.com is used as the repository URL instead of packagecloud.io. These URLs are our own URLs - and isolating users from packagecloud.io which is useful in case we need to change the packagecloud.io URL in the future. -2. Since our packages directly related to postgresql, we install postgresql repository for convenience. -3. Programs like curl, wget, gnupg, apt-transport-https are installed if they are not already installed which are required for installing postgres and citus packages. +However, custom installation scripts are provided for convenience. +The benefits of using the custom installation scripts are: +1. `install.citusdata.com/repos.citusdata.com` is used as the repository URL instead of packagecloud.io. These URLs are our own URLs, isolating users from packagecloud.io, which is useful in case we need to change the packagecloud.io URL in the future. +2. Since our packages are directly related to PostgreSQL, we install the PostgreSQL repository for convenience. +3. Programs like curl, wget, gnupg, and apt-transport-https are installed if they are not already installed, as they are required for installing PostgreSQL and Citus packages. Repository Installation Scripts are stored in the `packaging/gh-pages branch `_. -Files stored in this branch are served using GH pages. -You can see the GH Pages settings in the `settings `_. -The original url of the files are `https://citusdata.github.io/packaging/`_. -For example, the url of the `deb.sh < https://citusdata.github.io/packaging/community/deb.sh>`_. -install.citusdata.com/repos.citusdata.com is a CNAME record for citusdata.github.io in our DNS settings in Cloudflare. -For more information about url redirection, please refer to Citus Packaging Web Url Certificates document - -There are 8 scripts in this branch for 4 repositories the packagecloud.io. +Files stored in this branch are served using GH Pages. You can see the GH Pages settings in the `settings `_. +The original URL prefix of the files is `https://citusdata.github.io/packaging/`. +For example, the URL of `deb.sh `_. `install.citusdata.com/repos.citusdata.com` +is a CNAME record for `citusdata.github.io` in our DNS settings in Cloudflare. +For more information about URL redirection, please refer to the Citus Packaging Web Url Certificates document. + +There are 8 scripts in this branch for 4 repositories on packagecloud.io: + 1. community - * `deb.sh `_ - * `rpm.sh `_ + + - `Community deb.sh `_ + + - `Community rpm.sh `_ + 2. community-nightlies - * `deb.sh `_ - * `rpm.sh `_ + + - `community-nightlies deb.sh `_ + + - `community-nightlies rpm.sh `_ + 3. enterprise - * `deb.sh `_ - * `rpm.sh `_ + + - `enterprise deb.sh `_ + + - `enterprise rpm.sh `_ + 4. enterprise-nightlies - * `deb.sh `_ - * `rpm.sh `_ + + - `enterprise-nightlies deb.sh `_ + + - `enterprise-nightlies rpm.sh `_ These scripts are used to install the repositories. For example, to install the community repository, you can run the following command: -For Debian/Ubuntu for community repository: +For Debian/Ubuntu: -``` -curl https://install.citusdata.com/community/deb.sh | sudo bash -``` +.. code-block:: shell -For RHEL/CentOS for community repository: + curl https://install.citusdata.com/community/deb.sh | sudo bash -``` -curl https://install.citusdata.com/community/rpm.sh | sudo bash -``` +For RHEL/CentOS: +.. code-block:: shell + curl https://install.citusdata.com/community/rpm.sh | sudo bash From bddb8e9df1749010fad16381b1e35f8f735daba9 Mon Sep 17 00:00:00 2001 From: gindibay Date: Fri, 30 Jun 2023 13:29:05 +0300 Subject: [PATCH 03/11] Updates docs content --- source/citus-docker-images.rst | 88 +++++++++------------------------- 1 file changed, 23 insertions(+), 65 deletions(-) diff --git a/source/citus-docker-images.rst b/source/citus-docker-images.rst index 48b1f7f60..57e94bf2c 100644 --- a/source/citus-docker-images.rst +++ b/source/citus-docker-images.rst @@ -3,77 +3,35 @@ Citus Docker images General Structure ------------------ +Docker images are part of the Citus release process. They are built and pushed after every release. Dockerfiles are stored in the `citusdata/docker `_ repository. In each release, we change the below Dockerfiles to point to the new Citus release. -Docker images are part of the Citus release process. They are built and pushed after every release. -Dockerfiles are stored in `citusdata/docker `_ repository. -In each release, we change the below Dockerfiles to point to the new Citus release. -1. `Main Dockerfile `_ -which is used to build the latest regular Citus image for the latest Postgres version. -2. `alpine Dockerfile `_ -which is used to build the latest regular Citus image for the latest Postgres version -with the alpine base docker image. -3. `postgres-xx Dockerfiles which are used to build the Citus images for the older Postgres versions. -Normally, we support the last 3 major Postgres versions. For example, if the latest Postgres version is 15, -we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported postgres major versions other than the latest one. -For example, if we support 13, 14, and 15, we need to have postgres-13/Dockerfile -and postgres-14/Dockerfile Dockerfiles in our repository. +1. `Main Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version. +2. `alpine Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version with the alpine base Docker image. +3. `postgres-xx` Dockerfiles (where `xx` represents the major Postgres versions>`_: Used to build the Citus images for the older Postgres versions. We normally support the last 3 major Postgres versions. For example, if the latest Postgres version is 15, we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported Postgres major versions other than the latest one. Build Process ------------------ +The Dockerfiles are built and pushed to Docker Hub by GH Actions pipelines. There are four pipelines used for the Citus Docker image build and publish process: -The Dockerfiles are built and pushed to Docker Hub by GH Actions pipelines. -There are two pipelines to be used for Citus docker image build and publish process. -1. `Update Citus versions `_: -This pipeline should be triggered manually after finishing publishing of Citus packages to the packagecloud, -since the pipeline uses the Citus packages from the packagecloud to build the docker images. -This pipeline updates the Citus versions in the Dockerfiles and pushes the changes -to a newly created branch and creates a PR. -2. `Build and publish Citus docker images on push `_: -This pipeline is triggered on every branch push to repository. -This pipeline builds the docker images on every occasion and pushes them to Docker Hub if the branch is master. -Therefore, this pipeline is used to validate the build in the branches other than master and to publish the images to Docker Hub in the master branch. -3. `Build and publish Citus docker images on schedule `_: -This pipeline is triggered every day at 00:30 UTC and build and publish the docker images for -main, alpine and postgres-xx docker images to Docker Hub. -4. `Build and publish Citus docker images manually `_: -This pipeline is triggered manually and build and publish the docker images. It is useful when we want to build and publish the docker images -without waiting for the scheduled pipeline to run. - -New Postgres Version Support ------------------- - -When a new Postgres version is released, we need to add the new Dockerfile for the new Postgres version. -Steps to add a new Postgres version support: -1. Create a new branch from master branch. -2. Create a new Dockerfile for the new Postgres version. For example for postgres 16 we need to add postgres-16/Dockerfile by copying the postgres-15/Dockerfile. -3. Change the postgres version in the new Dockerfile to the new Postgres version. -4. Test the new Dockerfile by running the below command in the root directory of the repository. - -``` -docker build -t citusdata/citus:16.0.0-postgres-16 . -``` -5. If the build is successful, test the new docker image by running the below command. - -``` -# run PostgreSQL with Citus on port 5500 -docker run -d --name citus -p 5500:5432 -e POSTGRES_PASSWORD=mypassword citusdata/citus:16.0.0-postgres-16 -# connect using psql within the Docker container -docker exec -it citus psql -U postgres -# or, connect using local psql -psql -U postgres -d postgres -h localhost -p 5500 -``` -6. Add new postgres version support to the scheduled pipeline by adding the new postgres version to the pipelines. -7. If the tests are successful, create a PR to merge the changes to master branch. - -Adding new postgres version support to the tools scripts --------------------------------------------------------- - -When a new Postgres version is released, we need to add the new Postgres version support to the tools scripts. -Mainly `update_docker.py script -needs to be updated to add the new Postgres version support. -This script is being used in the pipelines. - +1. `Update Citus versions `_: This pipeline updates the Citus versions in the Dockerfiles and pushes the changes to a newly created branch and creates a PR. It should be triggered manually after finishing the publishing of Citus packages to the packagecloud. +2. `Build and publish Citus Docker images on push `_: This pipeline is triggered on every branch push to the repository. It builds the Docker images on every occasion and pushes them to Docker Hub if the branch is master. +3. `Build and publish Citus Docker images on schedule `_: This pipeline is triggered every day at 00:30 UTC and builds and publishes the Docker images for the main, alpine, and `postgres-xx` Docker images to Docker Hub. +4. `Build and publish Citus Docker images manually `_: This pipeline is triggered manually and builds and publishes the Docker images. It is useful when we want to build and publish the Docker images without waiting for the scheduled pipeline to run. +New Postgres Version Support +----------------------------- +When a new Postgres version is released, we need to add support for the new Postgres version in the Dockerfiles. To add support for a new Postgres version: + +1. Create a new branch from the master branch. +2. Create a new Dockerfile for the new Postgres version. For example, for Postgres 16, create `postgres-16/Dockerfile` by copying the `postgres-15/Dockerfile`. +3. Change the Postgres version in the new Dockerfile to the new Postgres version. +4. Test the new Dockerfile using the appropriate Docker build and run commands. +5. Add new Postgres version support to the scheduled pipeline. +6. If the tests are successful, create a pull request to merge the changes to the master branch. + +Adding new Postgres version support to the tools scripts +--------------------------------------------------------- +When a new Postgres version is released, we also need to add support for the new Postgres version in the tools scripts. Mainly, the `update_docker.py `_ script needs to be updated to add support for the new Postgres version. This script is used in the pipelines. From 527dbb6e6f89577f01f503c4955c9927f4a1a8d2 Mon Sep 17 00:00:00 2001 From: gindibay Date: Fri, 30 Jun 2023 15:10:45 +0300 Subject: [PATCH 04/11] Updates docs content --- .readthedocs.yaml | 15 ++ source/debian-packages.rst | 42 +++-- source/index.rst | 35 ++-- source/introduction.rst | 2 +- source/packaging-process.rst | 187 ++++++++++---------- source/pgdg.rst | 8 +- source/pgxn.rst | 19 +- source/project-packaging-configurations.rst | 57 +++--- source/repository-installation-scripts.rst | 29 +-- source/tools-scripts.rst | 28 ++- 10 files changed, 227 insertions(+), 195 deletions(-) create mode 100644 .readthedocs.yaml diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 000000000..f994c51b2 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,15 @@ +version: 2 + +build: + os: "ubuntu-22.04" + tools: + python: "3.11" + +# Build from the docs/ directory with Sphinx +sphinx: + configuration: source/conf.py + +# Explicitly set the version of Python and its requirements +python: + install: + - requirements: requirements.txt \ No newline at end of file diff --git a/source/debian-packages.rst b/source/debian-packages.rst index 3d975ba1a..4be76e8bd 100644 --- a/source/debian-packages.rst +++ b/source/debian-packages.rst @@ -1,31 +1,30 @@ -Package Preparation -==================== +.. _debian-packages: Debian Packages ----------------- +=============== Introduction to Debian packaging -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +-------------------------------- -Debian packaging provides a standardized way to create software packages for Debian-based operating systems. It ensures easy installation, upgrade, and removal of software components. The packaging format used in Debian is known as the Debian package format or `.deb`. +Debian packaging provides a standardized way to create software packages for Debian-based operating systems. It ensures easy installation, upgrade, and removal of software components. The packaging format used in Debian is known as the Debian package format or ``.deb``. Debian-specific tools and conventions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------- Debian packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following: -- **dpkg**: The Debian package manager (`dpkg`) is the core tool for installing, querying, and managing Debian packages. It operates at a lower level than higher-level package management tools like `apt` or `apt-get`. +- **dpkg**: The Debian package manager (``dpkg``) is the core tool for installing, querying, and managing Debian packages. It operates at a lower level than higher-level package management tools like ``apt`` or ``apt-get``. -- **Debian control file**: The `debian/control` file is a metadata file that provides information about the package, its dependencies, and other package-related details. It includes fields such as package name, version, maintainer, description, dependencies, and more. Maintaining an accurate and up-to-date control file is essential for proper package management. +- **Debian control file**: The ``debian/control`` file is a metadata file that provides information about the package, its dependencies, and other package-related details. It includes fields such as package name, version, maintainer, description, dependencies, and more. Maintaining an accurate and up-to-date control file is essential for proper package management. -- **Debian changelog**: The `debian/changelog` file maintains a record of changes made to the package across different versions. It includes information about the package version, the person responsible for the change, the change description, and other relevant details. The changelog helps users and maintainers track the history of the package and understand the modifications introduced in each release. +- **Debian changelog**: The ``debian/changelog`` file maintains a record of changes made to the package across different versions. It includes information about the package version, the person responsible for the change, the change description, and other relevant details. The changelog helps users and maintainers track the history of the package and understand the modifications introduced in each release. -- **Debian rules file**: The `debian/rules` file is a makefile-like script that defines the build process and other package-specific tasks. It specifies how to compile the source code, configure the package, and install the files in the correct locations. The rules file allows for customization and automation of the packaging process. +- **Debian rules file**: The ``debian/rules`` file is a makefile-like script that defines the build process and other package-specific tasks. It specifies how to compile the source code, configure the package, and install the files in the correct locations. The rules file allows for customization and automation of the packaging process. -Other important Debian-specific tools and files include `debuild`, `lintian`, and `pbuilder`, which aid in building, linting, and testing packages respectively. +Other important Debian-specific tools and files include ``debuild``, ``lintian``, and ``pbuilder``, which aid in building, linting, and testing packages respectively. Additional Debian Files -~~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- In addition to the core Debian packaging files, the packaging process may involve other files with specific purposes. Here are some of the additional Debian files commonly used: @@ -39,21 +38,20 @@ In addition to the core Debian packaging files, the packaging process may involv - **Upstream/signing-key.asc**: This file contains the ASCII-armored representation of the upstream signing key used to verify the authenticity and integrity of the package. -- **Changelog**: The `debian/changelog` file maintains a record of changes made to the package. +- **Changelog**: The ``debian/changelog`` file maintains a record of changes made to the package. -- **Compat**: The `debian/compat` file specifies the compatibility level of the package's packaging format. +- **Compat**: The ``debian/compat`` file specifies the compatibility level of the package's packaging format. -- **Control.in**: The `debian/control.in` file is a template file used to generate the `debian/control` file. +- **Control.in**: The ``debian/control.in`` file is a template file used to generate the ``debian/control`` file. -- **Copyright**: The `debian/copyright` file provides information about the copyright holders and licensing terms. +- **Copyright**: The ``debian/copyright`` file provides information about the copyright holders and licensing terms. -- **Docs**: The `debian/docs` file lists the additional documentation files included with the package. +- **Docs**: The ``debian/docs`` file lists the additional documentation files included with the package. -- **NOTICE**: The `debian/NOTICE` file contains any legally required notices or attributions. +- **NOTICE**: The ``debian/NOTICE`` file contains any legally required notices or attributions. -- **Pgversion**: The `debian/pgversion` file specifies the PostgreSQL version for which the package is intended. +- **Pgversion**: The ``debian/pgversion`` file specifies the PostgreSQL version for which the package is intended. -- **Rules**: The `debian/rules` file is a makefile-like script that defines the build process. +- **Rules**: The ``debian/rules`` file is a makefile-like script that defines the build process. - -For more information about Debian packaging, see the [Debian New Maintainers' Guide](https://www.debian.org/doc/manuals/maint-guide/). \ No newline at end of file +For more information about Debian packaging, see the `Debian New Maintainers' Guide `_. diff --git a/source/index.rst b/source/index.rst index bd9566144..b04f8182e 100644 --- a/source/index.rst +++ b/source/index.rst @@ -27,39 +27,44 @@ This documentation is intended to provide a detailed overview of the packaging p .Table of Contents =================== -Introduction ------------- + .. toctree:: - :maxdepth: 1 + :maxdepth: 2 introduction + + +.. toctree:: + :maxdepth: 2 + :caption: Repository Management + package-build-environment repository-installation-scripts -Tools and Packages ------------------- + .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Tools and Configurations tools-scripts - debian-packages - rpm-packages + project-packaging-configurations + -Citus Related -------------- .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Artifacts + debian-packages + rpm-packages citus-docker-images pgdg pgxn -Project Packaging ------------------ + .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Packaging Process - project-packaging-configurations packaging-process diff --git a/source/introduction.rst b/source/introduction.rst index 7444406dd..d00181907 100644 --- a/source/introduction.rst +++ b/source/introduction.rst @@ -1,5 +1,5 @@ Introduction -=============================== +============ Welcome to the documentation outlining the packaging process for Citus and its associated projects. Packaging plays a crucial role in the distribution and deployment of software, ensuring that it can be easily installed and utilized across various operating systems and environments. diff --git a/source/packaging-process.rst b/source/packaging-process.rst index 29dad2fc7..fd6db974c 100644 --- a/source/packaging-process.rst +++ b/source/packaging-process.rst @@ -1,7 +1,7 @@ Packaging Process ================= -Packaging process is done by the following steps: +The packaging process consists of the following steps: 1. Bake deb and rpm packages 2. Bake docker images @@ -11,140 +11,133 @@ Packaging process is done by the following steps: Bake deb and rpm packages --------------------------- -In this phase, we need to edit packaging configuration files to bake the packages for the desired application version. +In this phase, the packaging configuration files need to be edited to bake the packages for the desired application version. Editing Configuration Files by Pipeline ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Citus has its own automation to edit the packaging configuration scripts; `Update Package Properties `_ workflow. -After executing this workflow on all-citus branch, you will see a new PR to the packaging repository. This PR will update the packaging configuration files to the desired version. -After seeing all the checks are passed, you can request a review from the packaging team. -After the review is done, you can merge the PR. -After all jobs are done, you can see the new packages in the `Citusdata Package Repository `_. -You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. +Citus has an automation process to edit the packaging configuration scripts called the `Update Package Properties `_ workflow. + +By executing this workflow on the all-citus branch, a new pull request (PR) will be generated in the packaging repository. + +This PR will update the packaging configuration files to the desired version. + +Once the necessary checks have passed, a review can be requested from the packaging team. + +After the review is completed, the PR can be merged. + +Once all the jobs are finished, the new packages can be found in the `Citusdata Package Repository `_. + +It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. Editing Configuration Files Manually ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Other projects such as pg_azure_storage, topn does not have automation pipelines, so we need to edit the configuration files manually. -There are two categories of projects that we need to edit the configuration files manually. - -1. Projects that deb and rpm configuration files are in the same branch -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +For projects like pg_azure_storage and azure-gdpr that do not have automation pipelines, the configuration files need to be edited manually. +There are two categories of projects that require manual configuration file editing: -``pg_azure_storage``, ``azure-gdpr`` is in this category. -Below are the steps to edit the configuration files manually: +1. Projects with deb and rpm configuration files in the same branch: + - For projects in this category, such as pg_azure_storage and azure-gdpr, the following steps can be followed: + - Create a new branch from the latest commit on the master branch. + - Edit the configuration files to the desired version: + - Add a new entry to the `debian/changelog` file. + - Update the version in the pkgvars file. + - Update the version in the rpm spec file. + - Add a new changelog entry to the rpm spec file. + - Create a pull request (PR) from the new branch to the master branch. + - After the PR is merged, the new packages can be found in the `Citusdata Package Repository `_. It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. -1. Create a new branch from the latest commit on the master branch. -2. Edit the configuration files to the desired version. - * Add a new entry to the ``debian/changelog`` file. - * Update the version in the pkgvars file - * Update the version in the rpm spec file - * Add a new changelog entry to the rpm spec file -3. Create a PR from the new branch to the master branch. -4. After the PR is merged, you can see the new packages in the `Citusdata Package Repository `_. You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. +2. Projects with deb and rpm configuration files in different branches: + - For projects like topn, hll, and cron that have deb and rpm configuration files in different branches, the following steps can be followed: -2. Projects that deb and rpm configuration files are in different branches -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + **Debian Configuration Files:** + - Checkout the debian- branch. + - Create a new branch from the latest commit on the debian- branch. + - Edit the configuration files to the desired version: -``topn``, ``hll`` and ``cron`` is in this category. -Below are the steps to edit the configuration files manually: + - Add a new entry to the `debian/changelog` file. + - Update the version in the pkgvars file. -Debian Configuration Files -''''''''''''''''''''''''''' + - Create a PR from the new branch to the debian- branch. + - After the PR is merged, the new packages can be found in the `Citusdata Package Repository `_. + It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. -1. Checkout the debian- branch. -2. Create a new branch from the latest commit on the debian- branch. -3. Edit the configuration files to the desired version. - * Add a new entry to the ``debian/changelog`` file. - * Update the version in the pkgvars file -4. Create a PR from the new branch to the debian- branch. -5. After the PR is merged, you can see the new packages in the `Citusdata Package Repository `_. You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. + **Redhat Configuration Files:** + - Checkout the rpm- branch. + - Create a new branch from the latest commit on the rpm- branch. + - Edit the configuration files to the desired version: -Redhat Configuration Files -''''''''''''''''''''''''''' + - Update the version in the rpm spec file. + - Add a new changelog entry to the rpm spec file. -1. Checkout the rpm- branch. -2. Create a new branch from the latest commit on the rpm- branch. -3. Edit the configuration files to the desired version. - * Update the version in the rpm spec file - * Add a new changelog entry to the rpm spec file -4. Create a PR from the new branch to the rpm- branch. -5. After the PR is merged, you can see the new packages in the `Citusdata Package Repository `_. You need to make sure that the new packages are available for all distros we support in the repository before proceeding to the next step. + - Create a PR from the new branch to the rpm- branch. + - After the PR is merged, the new packages can be found in the `Citusdata Package Repository `_. It is important to ensure that the new packages are available for all supported distributions in the repository before proceeding to the next step. Bake docker images ------------------ -This step is applicable just for Citus Community. + +This step is applicable only for Citus Community. Baking Main versions ~~~~~~~~~~~~~~~~~~~~ -In this phase, we need to edit the docker image configuration files to bake the docker images for the desired application version. -To edit the docker image configuration files, you can use the `Update Version on Docker Files `_ workflow. -After executing this workflow on master branch, you will see a new PR to the docker repository. This PR will update the docker image configuration files to the desired version. -After seeing all the checks are passed, you can request a review from the packaging team. -After the review is done, you can merge the PR. -After all jobs are done, you can see the new docker images in the `Citusdata Docker Repository `_. -Versions that needs to be in the docker repository are: -1. latest -2. alpine -3. postgres_xx (all supported postgres versions before latest) -Create Images for Patch Releases -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +In this phase, the docker image configuration files need to be edited to bake the docker images for the desired application version. -In this phase, we need to create docker images for patch releases. For example, if we have a patch release for 10.1.1, we need to create a docker image for 10, 10.1 and 10.1.1. +The `Update Version on Docker Files `_ workflow can be used to edit the docker image configuration files. + +By executing this workflow on the master branch, a new pull request (PR) will be generated in the docker repository. + +This PR will update the docker image configuration files to the desired version. + +Once all the necessary checks have passed, a review can be requested from the packaging team. + +After the review is completed, the PR can be merged. + +Once all the jobs are finished, the new docker images can be found in the `Citusdata Docker Repository `_. -To create docker images for patch releases, you just need to add a tag to the latest commit on master branch and push it -`The tag pipeline `_ will create the docker images for the tag and push them to the docker repository. -After all jobs are done, you can see the new docker images in the `Citusdata Docker Repository `_. The versioned images should appear in the docker repository for all supported postgres versions and alpine. -For example for 10.1.1 for postgres 13,14,15; we need to see the following images in the docker repository: -1. 10.1.1 (latest i.e. 15) -2. 10.1 (latest i.e. 15) -3. 10 (latest i.e. 15) -4. 10.1.1-alpine -5. 10.1-alpine -6. 10-alpine -7. 10.1.1-pg13 -8. 10.1-pg13 -9. 10-pg13 -10. 10.1.1-pg14 -11. 10.1-pg14 -12. 10-pg14 +Create Images for Patch Releases +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In this phase, docker images need to be created for patch releases. +For example, if there is a patch release for version 10.1.1, docker images need to be created for versions 10, 10.1, and 10.1.1. -Bake pgxn packages --------------------------------------------- -When we build the pgxn extension for citus, we need to go through the following steps. +To create docker images for patch releases, a tag needs to be added to the latest commit on the master branch and pushed. -1. Execute the pipeline `Update Version on PGXN Config Files `_ to update the version in the pgxn configuration files. This pipeline will create a PR with the updated version. +The `The tag pipeline `_ pipeline will create the docker images for the tag and push them to the docker repository. -2. Check for the minimum postgres version. If it is different from the one that citus has, it should be changed to the one that citus has. +Once all the jobs are finished, the new docker images can be found in the `Citusdata Docker Repository `_. -3. Check for the tests and if they all pass, notify the PR reviewer to review the PR. +The versioned images should appear in the docker repository for all supported postgres versions and alpine. -4. Once the PR is reviewed, merge the PR and make sure that the `Citus PGXN `_ is updated with the new version. +Bake pgxn package +----------------- -Open issue for PGDG packaging --------------------------------------------- +When building the pgxn extension for Citus, the following steps need to be followed: -After all the packages are baked, we need to open an issue for PGDG packaging. -PGDG is a repository of PostgreSQL packages for several Linux distributions. -Citus packages are available for RPM based distributions. -When releasing Citus, we open an issue in Postgres Redmine to request a new release from the link below -https://redmine.postgresql.org/projects/pgrpms/issues/new -We select the fields as below: +1. Execute the `Update Version on PGXN Config Files `_ pipeline to update the version in the pgxn configuration files. This pipeline will create a pull request (PR) with the updated version. +2. Check for the minimum postgres version and update it if necessary. +3. Check that all the tests pass and notify the PR reviewer to review the PR. +4. Once the PR is reviewed, merge the PR and ensure that the Citus PGXN is updated with the new version. -* Tracker: Bug +Opening an issue for PGDG packaging +----------------------------------- -* Subject: https://redmine.postgresql.org/projects/pgrpms/issues/new +After all the packages are baked, an issue needs to be opened for PGDG packaging. PGDG is a repository of PostgreSQL packages for several Linux distributions. -* Description: Kindly release the following version of Citus: xx.x. +Citus packages are available for RPM-based distributions. +To request a new release for Citus, an issue needs to be opened in the `Postgres Redmine `_. -* Category: New Package +The required fields for the issue are as follows: -* Priority: Normal +- Tracker: Bug +- Subject: New release of Citus +- Description: Provide a detailed description of the new release, including any notable changes or improvements. +- Category: Packaging +- Priority: Normal +- Target version: +- Assignee: Leave blank unless specified -* Target version: +Please make sure to include all the necessary information and follow the bug reporting guidelines provided by the Postgres Redmine. -* Assignee: Devrim Gündüz \ No newline at end of file diff --git a/source/pgdg.rst b/source/pgdg.rst index 74e95f579..66f2db0f1 100644 --- a/source/pgdg.rst +++ b/source/pgdg.rst @@ -1,6 +1,8 @@ Citus PGDG Packages ==================== -PGDG is a repository of PostgreSQL packages for several Linux distributions. -Citus packages are available for RPM based distributions. -We don't own the process, we open the issue and Devrim Gunduz (PGDG owner) is taking care of it. \ No newline at end of file +Citus packages for RPM-based distributions are made available through the PGDG (PostgreSQL Global Development Group) repository. +PGDG is a repository that provides PostgreSQL packages for various Linux distributions. + +It is important to note that the packaging process for Citus packages in PGDG is not directly managed by us. +Instead, we open an issue in the PGDG repository and Devrim Gunduz, the owner of PGDG, takes care of the packaging process. diff --git a/source/pgxn.rst b/source/pgxn.rst index c189fe827..3176d4e13 100644 --- a/source/pgxn.rst +++ b/source/pgxn.rst @@ -3,26 +3,23 @@ Citus PGXN Extension Builds PGXN Build Infrastructure ---------------------------- -In packaging system, we have scripts and images to build PGXN extensions. -In package build environment, we have Docker images , so Dockerfiles, to build pgxn extensions. -`Docker image `_ -is the base image for all pgxn extensions. It has all the dependencies to build pgxn extensions. -In this docker image, there is a script called `fetch_and_build_pgxn `_ -which is used to build pgxn extensions. +In the packaging system, we have scripts and images to build PGXN extensions. +In the package build environment, we have Docker images, specifically Dockerfiles, to build pgxn extensions. + +The `Docker image `_ serves as the base image for all pgxn extensions. It contains all the dependencies required for building pgxn extensions. +Within this docker image, there is a script called `fetch_and_build_pgxn `_ which is used to build pgxn extensions. Citus PGXN Configuration ------------------------ -As each project package configuration, we have a branch named `pgxn-citus `_ -which is used to build pgxn extension for citus. In this branch, we have three configuration files used to build the packages +For each project package configuration, we maintain a branch named `pgxn-citus `_ specifically for building the pgxn extension for Citus. In this branch, we have three configuration files used for building the packages: -1. `META.json _ +1. `META.json `_ 2. pkgvars 3. postgres-matrix.yml -Pkgvars and postgres-matrix.yml has the same usage as in other project builds. +The usage of pkgvars and postgres-matrix.yml is similar to other project builds. .. _Pgxn Build Process: - diff --git a/source/project-packaging-configurations.rst b/source/project-packaging-configurations.rst index d7c707d13..4def845b1 100644 --- a/source/project-packaging-configurations.rst +++ b/source/project-packaging-configurations.rst @@ -1,42 +1,43 @@ Project Packaging Configurations -================================= +================================ + +For each project, a new branch is created in the packaging repository where packaging configuration files for the project are added. +The packaging configuration files include: -For each project, we open a new branch in the packaging repository. -In these branches, we add the packaging configuration files for the project. The packaging configuration files are: 1. Pkgvars 2. Postgres-matrix.yml Pkgvars ------- -Pkgvars is a property file that contains the basic information about the package. It is used by the packaging system to generate the package name, version, etc. +Pkgvars is a property file that contains essential information about the package. +It is used by the packaging system to generate the package name, version, and other details. + +Here is an example of a pkgvars file: -Below is a sample pkgvars file: +.. code-block:: ini -``` -pkgname=citus -hubproj=citus -pkgdesc='Citus (Open-Source)' -pkglatest=11.3.0.citus-1 -nightlyref=main -versioning=fancy -``` + pkgname=citus + hubproj=citus + pkgdesc='Citus (Open-Source)' + pkglatest=11.3.0.citus-1 + nightlyref=main + versioning=fancy Parameters for the pkgvars file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. pkgname: prefix of the package -2. hubproj: name of the repository in the GitHub -3. pkgdesc: description of the package. It is just for reference. It is not used by the packaging system -4. pkglatest: latest version of the package. It is used by the packaging system to generate the package name - and get the release number from the GitHub repository -5. nightlyref: branch name of the nightly builds -6. versioning: versioning scheme. It can be either `fancy` or `simple`. If it is `fancy`, - fancy version numbers are added to the package name. By default, it is `simple`. +1. pkgname: Prefix of the package. +2. hubproj: Name of the repository in GitHub. +3. pkgdesc: Description of the package. It is for reference purposes and not used by the packaging system. +4. pkglatest: Latest version of the package. It is used by the packaging system to generate the package name and retrieve the release number from the GitHub repository. +5. nightlyref: Branch name of the nightly builds. +6. versioning: Versioning scheme. It can be either 'fancy' or 'simple'. If 'fancy', fancy version numbers are added to the package name. The default is 'simple'. Postgres-matrix.yml ------------------- -Citus Postgres version matrix. Packaging system decides which postgres versions for the Citus version by referencing this file +The Postgres-matrix.yml file contains the Citus Postgres version matrix. +The packaging system utilizes this file to determine the appropriate Postgres versions for a given Citus version. -Below is a sample postgres-matrix.yml file: +Here is an example of a postgres-matrix.yml file: .. code-block:: yaml @@ -56,11 +57,9 @@ Below is a sample postgres-matrix.yml file: - 11.0: postgres_versions: [13, 14] - 11.1: - postgres_versions: [ 13, 14, 15 ] - -In packaging system, version number is taken from pkgvars file and correct matrix entry is being found from this file by comparing the version number with the matrix entries. -Then, the postgres versions are taken from the matrix entry and one package is generated for each postgres version. + postgres_versions: [13, 14, 15] -For example, if we want to create a package for 9.3.8 version of Citus, we will take the matrix entry for 9.0 and -create a package for each postgres version in the matrix entry since 9.3.8 is between 9.0 and 9.5. +In the packaging system, the version number is obtained from the pkgvars file, and the corresponding matrix entry is determined by comparing the version number with the entries in the matrix file. +The Postgres versions specified in the matrix entry are then used to generate individual packages. +For instance, if we want to create a package for Citus version 9.3.8, we will retrieve the matrix entry for version 9.0 and generate a package for each Postgres version listed in the matrix entry since 9.3.8 falls between 9.0 and 9.5. diff --git a/source/repository-installation-scripts.rst b/source/repository-installation-scripts.rst index cb6f858d7..ec5365537 100644 --- a/source/repository-installation-scripts.rst +++ b/source/repository-installation-scripts.rst @@ -1,18 +1,23 @@ Repository Installation Scripts =============================== -Citus packages are stored in packagecloud.io. To install the packages, you can use the packagecloud.io installation scripts directly. -However, custom installation scripts are provided for convenience. +Citus packages are hosted on packagecloud.io. You can install the packages directly using the packagecloud.io installation scripts. +However, we also provide custom installation scripts for convenience. The benefits of using the custom installation scripts are: -1. `install.citusdata.com/repos.citusdata.com` is used as the repository URL instead of packagecloud.io. These URLs are our own URLs, isolating users from packagecloud.io, which is useful in case we need to change the packagecloud.io URL in the future. -2. Since our packages are directly related to PostgreSQL, we install the PostgreSQL repository for convenience. -3. Programs like curl, wget, gnupg, and apt-transport-https are installed if they are not already installed, as they are required for installing PostgreSQL and Citus packages. - -Repository Installation Scripts are stored in the `packaging/gh-pages branch `_. -Files stored in this branch are served using GH Pages. You can see the GH Pages settings in the `settings `_. -The original URL prefix of the files is `https://citusdata.github.io/packaging/`. -For example, the URL of `deb.sh `_. `install.citusdata.com/repos.citusdata.com` -is a CNAME record for `citusdata.github.io` in our DNS settings in Cloudflare. + +1. The repository URL `install.citusdata.com/repos.citusdata.com` is used instead of packagecloud.io. + These URLs are managed by us, allowing isolation from potential changes to the packagecloud.io URL in the future. + +2. The PostgreSQL repository is automatically installed alongside our packages for ease of use. + +3. Essential programs such as curl, wget, gnupg, and apt-transport-https are installed if not already present, + as they are required for installing PostgreSQL and Citus packages. + +The Repository Installation Scripts are located in the `packaging/gh-pages branch `_. +These files are served using GH Pages, and the GH Pages settings can be found in the `settings `_. +The original URL prefix for the files is `https://citusdata.github.io/packaging/`. +For example, the URL for `deb.sh `_. +`install.citusdata.com/repos.citusdata.com` is a CNAME record for `citusdata.github.io` in our DNS settings on Cloudflare. For more information about URL redirection, please refer to the Citus Packaging Web Url Certificates document. There are 8 scripts in this branch for 4 repositories on packagecloud.io: @@ -41,7 +46,7 @@ There are 8 scripts in this branch for 4 repositories on packagecloud.io: - `enterprise-nightlies rpm.sh `_ -These scripts are used to install the repositories. For example, to install the community repository, you can run the following command: +These scripts are used to install the corresponding repositories. For example, to install the community repository, run the following command: For Debian/Ubuntu: diff --git a/source/tools-scripts.rst b/source/tools-scripts.rst index 75dfe28eb..b6b66696b 100644 --- a/source/tools-scripts.rst +++ b/source/tools-scripts.rst @@ -1,10 +1,28 @@ Packaging Tools Scripts ======================= -Packaging operations are performed by a set of python scripts stored under the `tools `_ repository. -The scripts are written in python and require python 3.8 or higher. The scripts are tested on Ubuntu focal. +Packaging operations are executed using a collection of Python scripts stored in the `tools `_ repository. +These scripts are written in Python and require Python 3.8 or higher. They have been tested on Ubuntu focal. -For each change we add a tag to the commit and push it to the repository. The tag is of the form `v..` format. -Mainly, we are using the scripts inside GH Actions pipelines. In these pipelines we use the tools scripts with the corresponding tag. +Whenever a change is made, a tag is added to the commit and pushed to the repository. The tag follows the `v..` format. +The scripts are primarily utilized within GH Actions pipelines, where the corresponding tag is used. -For more details on the tools scripts, please refer to the `tools README `_. \ No newline at end of file +The following scripts are employed for packaging purposes: + +Tools Scripts +------------- + +We utilize specific Python scripts to perform packaging operations. These scripts can be found at `https://github.com/citusdata/tools`. + +For each change, we create a new tag for the latest release and incorporate that tag into the pipelines. + +Packaging Scripts +~~~~~~~~~~~~~~~~~ + +The following provides an overview of the scripts used within the pipelines to execute various packaging operations: + +* citus_package: This is the main script responsible for building the packages. +* update_docker: This script generates Citus Docker images for end users. +* upload_to_package_cloud: Publishes the packages created by citus_package to Package Cloud. +* publish_docker: Publishes the Docker images created by the update_docker script. +* delete_packages_on_package_cloud: Removes nightly packages older than 10 days from Package Cloud to reduce repository size and lower Package Cloud bills. From 46845398454e6e6de9a13559ace91ecf8aefdf26 Mon Sep 17 00:00:00 2001 From: gindibay Date: Fri, 30 Jun 2023 15:18:20 +0300 Subject: [PATCH 05/11] Adds requirements.txt file --- requirements.txt | 62 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 requirements.txt diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 000000000..ec263df8f --- /dev/null +++ b/requirements.txt @@ -0,0 +1,62 @@ +# +# This file is autogenerated by pip-compile with Python 3.10 +# by the following command: +# +# pip-compile +# +alabaster==0.7.13 + # via sphinx +babel==2.12.1 + # via sphinx +certifi==2023.5.7 + # via requests +charset-normalizer==3.1.0 + # via requests +docutils==0.18.1 + # via + # sphinx + # sphinx-rtd-theme +idna==3.4 + # via requests +imagesize==1.4.1 + # via sphinx +jinja2==3.1.2 + # via sphinx +markupsafe==2.1.3 + # via jinja2 +packaging==23.1 + # via sphinx +pygments==2.15.1 + # via sphinx +readthedocs-sphinx-search==0.3.1 + # via -r requirements.in +requests==2.31.0 + # via sphinx +snowballstemmer==2.2.0 + # via sphinx +sphinx==6.2.1 + # via + # -r requirements.in + # sphinx-rtd-theme + # sphinxcontrib-jquery + # sphinxnotes-strike +sphinx-rtd-theme==1.2.2 + # via -r requirements.in +sphinxcontrib-applehelp==1.0.4 + # via sphinx +sphinxcontrib-devhelp==1.0.2 + # via sphinx +sphinxcontrib-htmlhelp==2.0.1 + # via sphinx +sphinxcontrib-jquery==4.1 + # via sphinx-rtd-theme +sphinxcontrib-jsmath==1.0.1 + # via sphinx +sphinxcontrib-qthelp==1.0.3 + # via sphinx +sphinxcontrib-serializinghtml==1.1.5 + # via sphinx +sphinxnotes-strike==1.2 + # via -r requirements.in +urllib3==2.0.3 + # via requests From 05eaad6b8841e8cafc81063fb76827105e12faaf Mon Sep 17 00:00:00 2001 From: gindibay Date: Fri, 30 Jun 2023 15:44:14 +0300 Subject: [PATCH 06/11] Fixes badges --- source/index.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/source/index.rst b/source/index.rst index b04f8182e..a01d4593c 100644 --- a/source/index.rst +++ b/source/index.rst @@ -2,15 +2,19 @@ Welcome to Citus Data documentation! ====================================================== -|Build Status| |Latest Documentation Status| +|Citus Linux Packages Nightly Status| |Citus Docker Image Nightly Status| |Latest Documentation Status| -.. |Build Status| image:: https://github.com/citusdata/activerecord-multi-tenant/actions/workflows/active-record-multi-tenant-tests.yml/badge.svg - :target: https://github.com/citusdata/activerecord-multi-tenant/actions/workflows/active-record-multi-tenant-tests.yml - :alt: Build Status +.. |Citus Linux Packages Nightly Status| image:: https://github.com/citusdata/packaging/actions/workflows/build-citus-community-nightlies.yml/badge.svg + :target: https://github.com/citusdata/packaging/actions/workflows/build-citus-community-nightlies.yml + :alt: Citus Linux Packages Nightly Status + +.. |Citus Docker Image Nightly Status| image:: https://github.com/citusdata/docker/actions/workflows/publish_docker_images_cron.yml/badge.svg + :target: https://github.com/citusdata/docker/actions/workflows/publish_docker_images_cron.yml + :alt: Citus Docker Image Nightly Status .. |Latest Documentation Status| image:: https://readthedocs.org/projects/django-multitenant/badge/?version=latest - :target: https://activerecord-multi-tenant.readthedocs.io/en/latest/?badge=latest + :target: https://citusdata-packaging.readthedocs.io/en/latest//?badge=latest :alt: Documentation Status From ed6be961f89cbd6262d64236e8c8c832334cfd46 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=BCrkan=20=C4=B0ndibay?= Date: Fri, 7 Jul 2023 18:24:35 +0300 Subject: [PATCH 07/11] Removes invalid characters in the link --- source/citus-docker-images.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/citus-docker-images.rst b/source/citus-docker-images.rst index 57e94bf2c..56f7fa2e4 100644 --- a/source/citus-docker-images.rst +++ b/source/citus-docker-images.rst @@ -7,7 +7,7 @@ Docker images are part of the Citus release process. They are built and pushed a 1. `Main Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version. 2. `alpine Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version with the alpine base Docker image. -3. `postgres-xx` Dockerfiles (where `xx` represents the major Postgres versions>`_: Used to build the Citus images for the older Postgres versions. We normally support the last 3 major Postgres versions. For example, if the latest Postgres version is 15, we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported Postgres major versions other than the latest one. +3. `postgres-xx` Dockerfiles (where `xx` represents the major Postgres versions) : Used to build the Citus images for the older Postgres versions. We normally support the last 3 major Postgres versions. For example, if the latest Postgres version is 15, we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported Postgres major versions other than the latest one. Build Process ------------------ From 54709507dd8578c2b55cf0758f3c7cdd2fa15e8d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?G=C3=BCrkan=20=C4=B0ndibay?= Date: Fri, 7 Jul 2023 18:29:10 +0300 Subject: [PATCH 08/11] Adds docker hub link --- source/citus-docker-images.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/source/citus-docker-images.rst b/source/citus-docker-images.rst index 56f7fa2e4..d7708aff8 100644 --- a/source/citus-docker-images.rst +++ b/source/citus-docker-images.rst @@ -4,6 +4,7 @@ Citus Docker images General Structure ------------------ Docker images are part of the Citus release process. They are built and pushed after every release. Dockerfiles are stored in the `citusdata/docker `_ repository. In each release, we change the below Dockerfiles to point to the new Citus release. +Citus Docker images are published in DockerHub `citusdata/citus `_ 1. `Main Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version. 2. `alpine Dockerfile `_: Used to build the latest regular Citus image for the latest Postgres version with the alpine base Docker image. From 5cdd3bac3a66b298a9f22f0ccea09098513b3821 Mon Sep 17 00:00:00 2001 From: gindibay Date: Tue, 11 Jul 2023 10:07:55 +0300 Subject: [PATCH 09/11] Adds repositories --- source/index.rst | 3 ++- source/repository.rst | 9 +++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 source/repository.rst diff --git a/source/index.rst b/source/index.rst index a01d4593c..cf8dc9fbd 100644 --- a/source/index.rst +++ b/source/index.rst @@ -1,5 +1,5 @@ -Welcome to Citus Data documentation! +Welcome to Citus Data Packaging documentation! ====================================================== |Citus Linux Packages Nightly Status| |Citus Docker Image Nightly Status| |Latest Documentation Status| @@ -44,6 +44,7 @@ This documentation is intended to provide a detailed overview of the packaging p package-build-environment repository-installation-scripts + repository .. toctree:: diff --git a/source/repository.rst b/source/repository.rst new file mode 100644 index 000000000..1bd439058 --- /dev/null +++ b/source/repository.rst @@ -0,0 +1,9 @@ +Package Repositories +===================== + +Baked packages are stored in various repositories. The repositories are: + +* `PackageCloud `_: Stores deb and rpm packages. Community users and Marlin uses this repository. This is the main repo for all users +* PGDG: Stores only latest Citus rpm packages. You can not find older versions here. We do not manage the packages here directly. We open a ticket with PGDG to create a new package. +* `PGXN `_: The PostgreSQL Extension Network (PGXN) is a community-driven effort aimed at providing a centralized platform for distributing and sharing extensions for the PostgreSQL database management system. We use this repository to store the citus extension. +* `Docker Hub `_: Stores docker images. We use this repository to store the docker images for citus. \ No newline at end of file From 0980f1b9f859c4d8070185f9a885c28217c9b8f4 Mon Sep 17 00:00:00 2001 From: gindibay Date: Tue, 11 Jul 2023 10:13:39 +0300 Subject: [PATCH 10/11] Fixes packagecloud link --- source/repository.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/repository.rst b/source/repository.rst index 1bd439058..67c43f8f8 100644 --- a/source/repository.rst +++ b/source/repository.rst @@ -3,7 +3,7 @@ Package Repositories Baked packages are stored in various repositories. The repositories are: -* `PackageCloud `_: Stores deb and rpm packages. Community users and Marlin uses this repository. This is the main repo for all users +* `PackageCloud `_: Stores deb and rpm packages. Community users and Marlin uses this repository. This is the main repo for all users * PGDG: Stores only latest Citus rpm packages. You can not find older versions here. We do not manage the packages here directly. We open a ticket with PGDG to create a new package. * `PGXN `_: The PostgreSQL Extension Network (PGXN) is a community-driven effort aimed at providing a centralized platform for distributing and sharing extensions for the PostgreSQL database management system. We use this repository to store the citus extension. * `Docker Hub `_: Stores docker images. We use this repository to store the docker images for citus. \ No newline at end of file From 8f93ef5d0cb0f0253e646c2443192eb40953cd57 Mon Sep 17 00:00:00 2001 From: gindibay Date: Tue, 11 Jul 2023 10:17:11 +0300 Subject: [PATCH 11/11] Fixes packagecloud link --- source/repository.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/repository.rst b/source/repository.rst index 67c43f8f8..c45dd7e22 100644 --- a/source/repository.rst +++ b/source/repository.rst @@ -3,7 +3,7 @@ Package Repositories Baked packages are stored in various repositories. The repositories are: -* `PackageCloud `_: Stores deb and rpm packages. Community users and Marlin uses this repository. This is the main repo for all users +* `PackageCloud `_: Stores deb and rpm packages. Community users and Marlin uses this repository. This is the main repo for all users * PGDG: Stores only latest Citus rpm packages. You can not find older versions here. We do not manage the packages here directly. We open a ticket with PGDG to create a new package. * `PGXN `_: The PostgreSQL Extension Network (PGXN) is a community-driven effort aimed at providing a centralized platform for distributing and sharing extensions for the PostgreSQL database management system. We use this repository to store the citus extension. * `Docker Hub `_: Stores docker images. We use this repository to store the docker images for citus. \ No newline at end of file