Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds packaging docs on readthedocs #1062

Open
wants to merge 11 commits into
base: packaging_documents
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Adds initial files
gurkanindibay committed Jun 30, 2023
commit 30928d2beb52a8ae3d2d74f573ee2254bb29d56a
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Ignore everything in this directory
.yardoc/*

28 changes: 28 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = source
BUILDDIR = build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# New target that executes the bash script before calling the original target
build-docs: pre-build # Change 'html' to the target you want to execute

pre-build:
# Execute your bash script here
bash api-reference.sh

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
html:
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

4 changes: 4 additions & 0 deletions requirements.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
sphinxnotes-strike
sphinx
sphinx_rtd_theme
readthedocs-sphinx-search
79 changes: 79 additions & 0 deletions source/citus-docker-images.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
Citus Docker images
====================

General Structure
------------------

Docker images are part of the Citus release process. They are built and pushed after every release.
Dockerfiles are stored in `citusdata/docker <https://github.com/citusdata/docker>`_ repository.
In each release, we change the below Dockerfiles to point to the new Citus release.
1. `Main Dockerfile <https://github.com/citusdata/docker/blob/master/Dockerfile>`_
which is used to build the latest regular Citus image for the latest Postgres version.
2. `alpine Dockerfile <https://github.com/citusdata/docker/tree/master/alpine>`_
which is used to build the latest regular Citus image for the latest Postgres version
with the alpine base docker image.
3. `postgres-xx Dockerfiles which are used to build the Citus images for the older Postgres versions.
Normally, we support the last 3 major Postgres versions. For example, if the latest Postgres version is 15,
we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported postgres major versions other than the latest one.
For example, if we support 13, 14, and 15, we need to have postgres-13/Dockerfile
and postgres-14/Dockerfile Dockerfiles in our repository.

Build Process
------------------

The Dockerfiles are built and pushed to Docker Hub by GH Actions pipelines.
There are two pipelines to be used for Citus docker image build and publish process.
1. `Update Citus versions <https://github.com/citusdata/docker/blob/master/.github/workflows/update_version.yml>`_:
This pipeline should be triggered manually after finishing publishing of Citus packages to the packagecloud,
since the pipeline uses the Citus packages from the packagecloud to build the docker images.
This pipeline updates the Citus versions in the Dockerfiles and pushes the changes
to a newly created branch and creates a PR.
2. `Build and publish Citus docker images on push <https://github.com/citusdata/docker/blob/master/.github/workflows/publish_docker_images_on_push.yml>`_:
This pipeline is triggered on every branch push to repository.
This pipeline builds the docker images on every occasion and pushes them to Docker Hub if the branch is master.
Therefore, this pipeline is used to validate the build in the branches other than master and to publish the images to Docker Hub in the master branch.
3. `Build and publish Citus docker images on schedule <https://github.com/citusdata/docker/blob/master/.github/workflows/publish_docker_images_cron.yml>`_:
This pipeline is triggered every day at 00:30 UTC and build and publish the docker images for
main, alpine and postgres-xx docker images to Docker Hub.
4. `Build and publish Citus docker images manually <https://github.com/citusdata/docker/blob/master/.github/workflows/publish_docker_images_on_manual.yml>`_:
This pipeline is triggered manually and build and publish the docker images. It is useful when we want to build and publish the docker images
without waiting for the scheduled pipeline to run.

New Postgres Version Support
------------------

When a new Postgres version is released, we need to add the new Dockerfile for the new Postgres version.
Steps to add a new Postgres version support:
1. Create a new branch from master branch.
2. Create a new Dockerfile for the new Postgres version. For example for postgres 16 we need to add postgres-16/Dockerfile by copying the postgres-15/Dockerfile.
3. Change the postgres version in the new Dockerfile to the new Postgres version.
4. Test the new Dockerfile by running the below command in the root directory of the repository.

```
docker build -t citusdata/citus:16.0.0-postgres-16 .
```
5. If the build is successful, test the new docker image by running the below command.

```
# run PostgreSQL with Citus on port 5500
docker run -d --name citus -p 5500:5432 -e POSTGRES_PASSWORD=mypassword citusdata/citus:16.0.0-postgres-16
# connect using psql within the Docker container
docker exec -it citus psql -U postgres
# or, connect using local psql
psql -U postgres -d postgres -h localhost -p 5500
```
6. Add new postgres version support to the scheduled pipeline by adding the new postgres version to the pipelines.
7. If the tests are successful, create a PR to merge the changes to master branch.

Adding new postgres version support to the tools scripts
--------------------------------------------------------

When a new Postgres version is released, we need to add the new Postgres version support to the tools scripts.
Mainly `update_docker.py <https://github.com/citusdata/tools/blob/develop/packaging_automation/update_docker.py> script
needs to be updated to add the new Postgres version support.
This script is being used in the pipelines.





30 changes: 30 additions & 0 deletions source/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
from datetime import date

# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = "Citus Packaging"
copyright = f"{date.today().year} .Citus Data Licensed under the MIT license, see License for details. "
author = "Citus Data"
release = "2.2.0"

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = ["sphinxnotes.strike"]

templates_path = ["_templates"]
exclude_patterns = []

language = "python"

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = "sphinx_rtd_theme"
html_static_path = ["_static"]
59 changes: 59 additions & 0 deletions source/debian-packages.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
Package Preparation
==================

Debian Packages
---------------

Introduction to Debian packaging
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Debian packaging provides a standardized way to create software packages for Debian-based operating systems. It ensures easy installation, upgrade, and removal of software components. The packaging format used in Debian is known as the Debian package format or `.deb`.

Debian-specific tools and conventions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Debian packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following:

- **dpkg**: The Debian package manager (`dpkg`) is the core tool for installing, querying, and managing Debian packages. It operates at a lower level than higher-level package management tools like `apt` or `apt-get`.

- **Debian control file**: The `debian/control` file is a metadata file that provides information about the package, its dependencies, and other package-related details. It includes fields such as package name, version, maintainer, description, dependencies, and more. Maintaining an accurate and up-to-date control file is essential for proper package management.

- **Debian changelog**: The `debian/changelog` file maintains a record of changes made to the package across different versions. It includes information about the package version, the person responsible for the change, the change description, and other relevant details. The changelog helps users and maintainers track the history of the package and understand the modifications introduced in each release.

- **Debian rules file**: The `debian/rules` file is a makefile-like script that defines the build process and other package-specific tasks. It specifies how to compile the source code, configure the package, and install the files in the correct locations. The rules file allows for customization and automation of the packaging process.

Other important Debian-specific tools and files include `debuild`, `lintian`, and `pbuilder`, which aid in building, linting, and testing packages respectively.

Additional Debian Files
~~~~~~~~~~~~~~~~~~~~~~

In addition to the core Debian packaging files, the packaging process may involve other files with specific purposes. Here are some of the additional Debian files commonly used:

- **Source/format**: This file specifies the format version of the Debian source package.

- **Source/lintian-overrides**: This file is used to override specific lintian warnings or errors generated during the package build process.

- **Tests/control**: This file contains information about the tests to be run on the package.

- **Tests/installcheck**: This file specifies commands or scripts to be executed after the package is installed.

- **Upstream/signing-key.asc**: This file contains the ASCII-armored representation of the upstream signing key used to verify the authenticity and integrity of the package.

- **Changelog**: The `debian/changelog` file maintains a record of changes made to the package.

- **Compat**: The `debian/compat` file specifies the compatibility level of the package's packaging format.

- **Control.in**: The `debian/control.in` file is a template file used to generate the `debian/control` file.

- **Copyright**: The `debian/copyright` file provides information about the copyright holders and licensing terms.

- **Docs**: The `debian/docs` file lists the additional documentation files included with the package.

- **NOTICE**: The `debian/NOTICE` file contains any legally required notices or attributions.

- **Pgversion**: The `debian/pgversion` file specifies the PostgreSQL version for which the package is intended.

- **Rules**: The `debian/rules` file is a makefile-like script that defines the build process.


For more information about Debian packaging, see the [Debian New Maintainers' Guide](https://www.debian.org/doc/manuals/maint-guide/).
40 changes: 40 additions & 0 deletions source/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
.. Django Multi-tenant documentation master file, created by
sphinx-quickstart on Mon Feb 13 13:32:28 2023.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to ActiveRecord Multi-tenant's documentation!
======================================================

|Build Status| |Latest Documentation Status|

.. |Build Status| image:: https://github.com/citusdata/activerecord-multi-tenant/actions/workflows/active-record-multi-tenant-tests.yml/badge.svg
:target: https://github.com/citusdata/activerecord-multi-tenant/actions/workflows/active-record-multi-tenant-tests.yml
:alt: Build Status


.. |Latest Documentation Status| image:: https://readthedocs.org/projects/django-multitenant/badge/?version=latest
:target: https://activerecord-multi-tenant.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status


Citus Data Packaging Documentation
========================================

Welcome to the official documentation for ``CitusData Packaging``.

.. toctree::
:maxdepth: 2
:caption: Table of Contents

introduction
package-build-environment
repository-installation-scripts
debian-packages
rpm-packages
tool-scripts
project-packaging-configurations
citus-docker-images
pgdg
pgxn

47 changes: 47 additions & 0 deletions source/introduction.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
Packaging Process Documentation
==============================

Introduction
------------

Welcome to the documentation outlining the packaging process for Citus and its associated projects. Packaging plays a crucial role in the distribution and deployment of software, ensuring that it can be easily installed and utilized across various operating systems and environments.

In this document, we will explore the steps and best practices involved in preparing packages for different operating systems and releases. We will focus on the Debian and RPM package formats, as they are widely used and supported in the industry. Additionally, we will cover the creation and management of Docker environments to facilitate packaging for specific OS/release/PostgreSQL version combinations.

The supported operating systems and releases for packaging include:

- CentOS 8
- CentOS 7
- Oracle Linux 8
- Oracle Linux 7
- AlmaLinux 9
- Debian Buster
- Debian Bullseye
- Debian Bookworm
- Ubuntu Bionic
- Ubuntu Focal
- Ubuntu Jammy
- Ubuntu Kinetic

The packaging process will involve preparing packages for various projects, including:

- `Citus <https://github.com/citusdata/packaging/tree/all-citus>`_
- `Pgazure <https://github.com/citusdata/packaging/tree/all-pg-azure-storage>`_
- Cron
- `Debian <https://github.com/citusdata/packaging/tree/debian-cron>`_
- `RPM <https://github.com/citusdata/packaging/tree/redhat-cron>`_
- Hll
- `Debian <https://github.com/citusdata/packaging/tree/debian-hll>`_
- `RPM <https://github.com/citusdata/packaging/tree/redhat-hll>`_
- Topn
- `Debian <https://github.com/citusdata/packaging/tree/debian-topn>`_
- `RPM <https://github.com/citusdata/packaging/tree/redhat-topn>`_
- `Azure-gdpr <https://github.com/citusdata/packaging/tree/all-azure_gdpr>`_

Once the packages are ready, we will push them to PackageCloud, a package management and distribution service. Additionally, we will create Docker images for each OS/release/PostgreSQL version combination, which will be published on Docker Hub.

Lastly, we will cover the publishing of the applications into PGXN (PostgreSQL Extension Network), a platform that hosts and distributes PostgreSQL extensions.

By following the steps outlined in this document, you will gain a comprehensive understanding of the packaging process for Citus and related projects. This knowledge will empower you to efficiently distribute and deploy the software across a wide range of operating systems and environments.

Now, let's dive into the detailed steps for package preparation, Docker environment management, Docker image creation, PackageCloud integration, and PGXN publishing.
97 changes: 97 additions & 0 deletions source/package-build-environment.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
III. Packaging Build Environment Management
---------------------------------

A. Introduction to Packaging environments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Mainly, we are using docker images for each OS/release/postgres version pair. We are using docker images to create a reproducible and consistent packaging environment. Docker images provide a reliable and consistent way to package software applications. They offer several advantages for packaging processes:
1. Isolation: Docker containers encapsulate applications and their dependencies, ensuring isolation from the underlying host system. This isolation helps prevent conflicts between different software components and provides a clean and reproducible environment for packaging.
2. Reproducibility: Docker environments enable the creation of reproducible packaging environments. By defining the dependencies and configurations within a Docker image, it becomes easier to recreate the same environment across different systems, ensuring consistent packaging results.
3. Portability: Docker environments are portable and can be easily shared across different systems. This portability ensures that the same packaging results can be achieved across different systems, eliminating environment-related issues.
4. Scalability: Docker environments can be easily scaled up or down to meet the packaging requirements. This scalability ensures that the packaging process can be easily scaled up or down to meet the packaging requirements.
5. Other: Docker environments provide a secure,reliable,consistent,flexible,efficient,simple and maintainable packaging environment. Docker environments provide this by isolating the packaging process from the underlying host system.

B. Docker images for various package types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We are using docker images for each OS/release/postgres version pair. We are using docker images to create a reproducible and consistent packaging environment.
For debian based packages, we can use the same docker image for all postgres versions. For example, we can use the same docker image for debian/10 postgres 13 and debian/10 postgres 14 packages.
For rpm based packages, we need a dedicated docker image for each postgres version. For example, we need a docker image for centos/7 postgres 13 and centos/7 postgres 14 packages.

Below is our supported os/release pair

1. Centos
1. centos/7
2. Oracle Linux
1. oraclelinux/7
2. oraclelinux/8
3. AlmaLinux
1. almalinux/9
4. Debian
1. debian/buster
2. debian/bullseye
3. debian/bookworm
5. Ubuntu
1. ubuntu/bionic
2. ubuntu/focal
3. ubuntu/jammy
4. ubuntu/kinetic
6. pgxn

Below is our supported postgres version
1. 13
2. 14
3. 15
4. 16 (planned)

Therefore, docker images for each OS/release/postgres version pair are as follows:

1. Centos
1. centos/7 postgres 13
2. centos/7 postgres 14
3. centos/7 postgres 15
4. centos/7 postgres 16 (planned)
2. Oracle Linux
1. oraclelinux/7 postgres 13
2. oraclelinux/7 postgres 14
3. oraclelinux/7 postgres 15
4. oraclelinux/7 postgres 16 (planned)
5. oraclelinux/8 postgres 13
6. oraclelinux/8 postgres 14
7. oraclelinux/8 postgres 15
8. oraclelinux/8 postgres 16 (planned)
3. AlmaLinux
1. almalinux/9 postgres 13
2. almalinux/9 postgres 14
3. almalinux/9 postgres 15
4. almalinux/9 postgres 16 (planned)
4. Debian
1. debian/buster all postgres versions
2. debian/bullseye all postgres versions
3. debian/bookworm all postgres versions
5. Ubuntu
1. ubuntu/bionic all postgres versions
2. ubuntu/focal all postgres versions
3. ubuntu/jammy all postgres versions
4. ubuntu/kinetic all postgres versions
6. Pgxn

C. Developing and Maintaining Docker Images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Dockerfiles, which are used to build docker images, are located in the develop branch of packaging repository.
We have a template structure to auto-generate the docker files for each OS/release/postgres version pair. The template structure is as follows:

1. `Dockerfile-deb.tmpl <https://github.com/citusdata/packaging/blob/develop/templates/Dockerfile-deb.tmpl>`_: This template is used to generate docker files for debian based docker files.
2. `Dockerfile-rpm.tmpl <https://github.com/citusdata/packaging/blob/develop/templates/Dockerfile-rpm.tmpl>`_ : This template is used to generate docker files for rpm based docker files.
3. `Dockerfile-pgxn.tmpl <https://github.com/citusdata/packaging/blob/develop/templates/Dockerfile-pgxn.tmpl>`_ : This template is used to generate docker files for pgxn based docker files.

After changing the template files, we need to run the `update_dockerfiles.sh <https://github.com/citusdata/packaging/blob/develop/update-dockerfiles.sh>`_ script to generate the docker files for each OS/release/postgres version pair.
After commit and push the changes, GH actions will build the docker images using the generated docker files. After seeing all the docker images are built successfully, we can merge the changes to the master branch.
When we merge the changes to the master branch, GH actions will push the docker images to the `citus/packaging <https://hub.docker.com/r/citus/packaging>`_ docker hub repository.

If you want to publish test images from `citus/packaging-test <https://hub.docker.com/r/citus/packaging-test>`_, you can use the `test pipeline <https://github.com/citusdata/packaging/blob/develop/.github/workflows/build-package-test.yml>`_.
To use the test pipeline, you need to change the current branch into your branch name. In this case, the test pipeline will push the docker images to the `citus/packaging-test <https://hub.docker.com/r/citus/packaging-test>`_ docker hub repository.



15 changes: 15 additions & 0 deletions source/pgdg.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
Citus PGDG Packages
====================

PGDG is a repository of PostgreSQL packages for several Linux distributions.
Citus packages are available for RPM based distributions.
When releasing Citus, we open an issue in Postgres Redmine to request a new release from the link below
https://redmine.postgresql.org/projects/pgrpms/issues/new
We select the fields as below:
* Tracker: Bug
* Subject: https://redmine.postgresql.org/projects/pgrpms/issues/new
* Description: Kindly release the following version of Citus: xx.x.
* Category: New Package
* Priority: Normal
* Target version: <The version of PostgreSQL that Citus is compatible with>
* Assignee: Devrim Gündüz
35 changes: 35 additions & 0 deletions source/pgxn.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Citus PGXN Extension Builds
===========================

PGXN Build Infrastructure
----------------------------
In packaging system, we have scripts and images to build PGXN extensions.
In package build environment, we have Docker images , so Dockerfiles, to build pgxn extensions.
`Docker image <https://github.com/citusdata/packaging/blob/develop/dockerfiles/pgxn-all/Dockerfile>`_
is the base image for all pgxn extensions. It has all the dependencies to build pgxn extensions.
In this docker image, there is a script called `fetch_and_build_pgxn <https://github.com/citusdata/packaging/blob/develop/scripts/fetch_and_build_pgxn>`_
which is used to build pgxn extensions.

Citus PGXN COnfiguration
------------------------

As each project package configuration, we have a branch named `pgxn-citus <https://github.com/citusdata/packaging/tree/pgxn-citus>`_
which is used to build pgxn extension for citus. In this branch, we have three configuration files used to build the packages
1. `META.json <https://pgxn.org/spec/`>_
2. pkgvars
3. postgres-matrix.yml

Pkgvars and postgres-matrix.yml has the same usage as in other project builds.

Building PGXN Extension for the new release
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When we build the pgxn extenson for citus, we need to go through the following steps.
1. Execute the pipeline `Update Version on PGXN Config Files <https://github.com/citusdata/packaging/actions/workflows/update-pgxn-version.yml>`_
to update the version in the pgxn configuration files. This pipeline will create a PR with the updated version.
2. Check for the minimum postgres version. If it is different from the one that citus has, it should be changed to the one that citus has.
3. Check for the tests and if they all pass, notify the PR reviewer to review the PR.
4. Once the PR is reviewed, merge the PR and make sure that the `Citus PGXN <https://pgxn.org/dist/citus/>`_ is updated with the new version.




69 changes: 69 additions & 0 deletions source/project-packaging-configurations.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
Project Packaging Configurations
=================================

For each project, we open a new branch in the packaging repository.
In these branches, we add the packaging configuration files for the project. The packaging configuration files are:
1. Pkgvars
2. Postgres-matrix.yml

Pkgvars
-------
Pkgvars is a property file that contains the basic information about the package. It is used by the packaging system to generate the package name, version, etc.

Below is a sample pkgvars file:

```
pkgname=citus
hubproj=citus
pkgdesc='Citus (Open-Source)'
pkglatest=11.3.0.citus-1
nightlyref=main
versioning=fancy

```

Parameters for the pkgvars file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. pkgname: prefix of the package
2. hubproj: name of the repository in the GitHub
3. pkgdesc: description of the package. It is just for reference. It is not used by the packaging system
4. pkglatest: latest version of the package. It is used by the packaging system to generate the package name and get the release number
from the GitHub repository
5. nightlyref: branch name of the nightly builds
6. versioning: versioning scheme. It can be either `fancy` or `simple`. If it is `fancy`,fancy version numbers are added to the package name.
By default, it is `simple`.

Postgres-matrix.yml
-------
Citus Postgres version matrix. Packaging system decides which postgres versions for the Citus version by referencing this file

Below is a sample postgres-matrix.yml file:

```
name: Postgres Version Matrix
project_name: citus

version_matrix:
- 8.0:
postgres_versions: [10, 11]
- 9.0:
postgres_versions: [11, 12]
- 9.5:
postgres_versions: [11, 12, 13]
- 10.1:
postgres_versions: [12, 13]
- 10.2:
postgres_versions: [12, 13, 14]
- 11.0:
postgres_versions: [13, 14]
- 11.1:
postgres_versions: [ 13, 14, 15 ]

```

In packaging system, version number is taken from pkgvars file and correct matrix entry is being found from this file by comparing the version number with the matrix entries.
Then, the postgres versions are taken from the matrix entry and one package is generated for each postgres version.

For example, if we want to create a package for 9.3.8 version of Citus, we will take the matrix entry for 9.0 and
create a package for each postgres version in the matrix entry since 9.3.8 is between 9.0 and 9.5.

47 changes: 47 additions & 0 deletions source/repository-installation-scripts.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
Repository Installation Scripts
===============================

Citus packages are stored in packagecloud.io. To install the packages, you can use the packagecloud.io installation scripts directly.
However, custom installation scripts are provided for convenience. Benefits of using the custom installation scripts are:
1. install.citusdata.com/repos.citusdata.com is used as the repository URL instead of packagecloud.io. These URLs are our own URLs
and isolating users from packagecloud.io which is useful in case we need to change the packagecloud.io URL in the future.
2. Since our packages directly related to postgresql, we install postgresql repository for convenience.
3. Programs like curl, wget, gnupg, apt-transport-https are installed if they are not already installed which are required for installing postgres and citus packages.

Repository Installation Scripts are stored in the `packaging/gh-pages branch <https://github.com/citusdata/packaging/tree/gh-pages>`_.
Files stored in this branch are served using GH pages.
You can see the GH Pages settings in the `settings <https://github.com/citusdata/packaging/settings/pages>`_.
The original url of the files are `https://citusdata.github.io/packaging/<path to file>`_.
For example, the url of the `deb.sh < https://citusdata.github.io/packaging/community/deb.sh>`_.
install.citusdata.com/repos.citusdata.com is a CNAME record for citusdata.github.io in our DNS settings in Cloudflare.
For more information about url redirection, please refer to Citus Packaging Web Url Certificates document

There are 8 scripts in this branch for 4 repositories the packagecloud.io.
1. community
* `deb.sh <https://github.com/citusdata/packaging/blob/gh-pages/community/deb.sh>`_
* `rpm.sh <https://github.com/citusdata/packaging/blob/gh-pages/community/rpm.sh>`_
2. community-nightlies
* `deb.sh <https://github.com/citusdata/packaging/blob/gh-pages/community-nightlies/deb.sh>`_
* `rpm.sh <https://github.com/citusdata/packaging/blob/gh-pages/community-nightlies/rpm.sh>`_
3. enterprise
* `deb.sh <https://github.com/citusdata/packaging/blob/gh-pages/enterprise/deb.sh>`_
* `rpm.sh <https://github.com/citusdata/packaging/blob/gh-pages/enterprise/rpm.sh>`_
4. enterprise-nightlies
* `deb.sh <https://github.com/citusdata/packaging/blob/gh-pages/enterprise-nightlies/deb.sh>`_
* `rpm.sh <https://github.com/citusdata/packaging/blob/gh-pages/enterprise-nightlies/rpm.sh>`_

These scripts are used to install the repositories. For example, to install the community repository, you can run the following command:

For Debian/Ubuntu for community repository:

```
curl https://install.citusdata.com/community/deb.sh | sudo bash
```

For RHEL/CentOS for community repository:

```
curl https://install.citusdata.com/community/rpm.sh | sudo bash
```


30 changes: 30 additions & 0 deletions source/rpm-packages.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
RPM Packages
------------

Introduction to RPM packaging
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

RPM packaging is a common method used to create software packages for RPM-based Linux distributions. RPM (Red Hat Package Manager) provides a reliable and consistent way to install, upgrade, and remove software components. The packaging format used in RPM is known as the RPM package format or `.rpm`.

RPM-specific tools and conventions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

RPM packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following:

- **rpmbuild**: The `rpmbuild` command is the primary tool for building RPM packages. It compiles the source code, creates the package structure, and generates the final RPM package.

- **.spec file**: The `.spec` file is a key component of RPM packaging. It contains detailed instructions and metadata about the package, such as its name, version, release, dependencies, build requirements, and file locations. The `.spec` file defines the build process, file ownership, permissions, and other package-specific configurations.

- **rpmlintrc**: The `rpmlintrc` file is used to configure the rpmlint tool, which performs static analysis on the generated RPM package. The rpmlint tool checks for common packaging issues, compliance with guidelines, and potential errors. The `rpmlintrc` file allows you to specify specific checks to enable or disable for your package.

Additional RPM Files
~~~~~~~~~~~~~~~~~~~~

In addition to the core RPM packaging components, the packaging process may involve other files with specific purposes. Here are some additional RPM-related files commonly used:

- **.spec file**: The `.spec` file contains the build instructions and metadata for the RPM package.

- **rpmlintrc**: The `rpmlintrc` file configures the rpmlint tool for checking the package.

For more information about RPM packaging, see the [RPM Packaging Guide](https://rpm-packaging-guide.github.io/).

10 changes: 10 additions & 0 deletions source/tools-scripts.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Packaging Tools Scripts
=======================

Packaging operations are performed by a set of python scripts stored under the `tools <https://github.com/citusdata/tools>`_ repository.
The scripts are written in python and require python 3.8 or higher. The scripts are tested on Ubuntu focal.

For each change we add a tag to the commit and push it to the repository. The tag is of the form `v<major_version>.<minor_version>.<patch_version>` format.
Mainly, we are using the scripts inside GH Actions pipelines. In these pipelines we use the tools scripts with the corresponding tag.

For more details on the tools scripts, please refer to the `tools README <https://github.com/citusdata/tools/blob/master/README.md>`_.