arenadata · Stolb27 · Dec 7, 2023 · Sep 11, 2023 · Sep 12, 2023 · Sep 12, 2023
diff --git a/.abi-check/.images/download-report-from-gh-action.png b/.abi-check/.images/download-report-from-gh-action.png
diff --git a/.abi-check/6.25.3/postgres.symbols.ignore b/.abi-check/6.25.3/postgres.symbols.ignore
@@ -0,0 +1,2 @@
+DummySymbol
+ConfigureNamesInt_gp
diff --git a/.abi-check/6.25.3/postgres.types.ignore b/.abi-check/6.25.3/postgres.types.ignore
@@ -0,0 +1 @@
+DummyType
diff --git a/.abi-check/README.md b/.abi-check/README.md
@@ -0,0 +1,74 @@
+# Check the compatibility of Greenplum ABI.
+
+## Introduction
+
+We use the [`abi-dumper`](https://github.com/lvc/abi-dumper) and [`abi-compliance-checker`](https://github.com/lvc/abi-compliance-checker/) to check the Greenplum's ABI. We also use the [GitHub action](../.github/workflows/greenplum-abi-tests.yml) to automate this job.
+
+## Requirements
+
+`abi-dumper` requires the binary being compiled with `-Og -g3`, hence the `CFLAGS` for configuration looks like:
+
+```bash
+## GCC's maybe-uninitialized checker may produce false positives with different
+## levels of optimizations. To prevent building failures, we append the '-Wno-maybe-uninitialized'
+## to the $CFLAGS as well.
+CFLAGS='-Og -g3 -Wno-maybe-uninitialized' ./configure --with-xxx --with-yyy --with-zzz
+```
+
+## Check the ABI's compatibility
+
+Several binaries are shipped in Greenplum, e.g., `$GPHOME/bin/postgres`, `$GPHOME/lib/libpq.so`, etc. Since the `postgres` binary are referenced by many extensions, the ABI compatibility of it is the most important. The following steps illustrate how to check the ABI compatibility of the `postgres` binary.
+
+1. Dump the ABI information of one `postgres` binary.
+   ```
+   abi-dumper $GPHOME/bin/postgres -lver <version of the binary> -o <output file>
+   ```
+   - `<version of the binary>`: The version of the binary. You can give it some reasonable name, e.g., `6.25.3` to indicate the binary is built from '6.25.3' tag.
+   - `<output file>`: The file path for dumping the ABI information, e.g., `greenplum-6.25.3.dump`
+
+2. Dump the ABI information of another `postgres` binary (same as the step 1).
+
+3. Compare the ABI between these two binaries with `abi-compliance-checker`.
+   ```
+   abi-compliance-checker \
+         -lib <library name> \
+         -old <output file from step1> \
+         -new <output file from step2>
+   ```
+   - `<library name>`: The name of the library, e.g., `postgres`.
+
+4. By default, the `abi-compliance-checker` will produce an HTML web page and there will be detailed information about ABI changes.
+
+## Ignore the "Safe ABI breaking change"
+
+There might be "safe ABI breaking changes", e.g., some symbol being removed and not referenced by any extensions or programs. Here are steps on how to suppress such errors.
+
+1. Add ignored symbols to `gpdb_src/.abi-check/<base version>/postgres.symbols.ignore` (one symbol per line).
+   - `<base version>`: The baseline version of Greenplum. If we want to ensure the ABI isn't broken between the `6.25.3` release and the latest `6X_STABLE`. The baseline version of Greenplum is `6.25.3`. See: [./6.25.3/postgres.symbols.ignore](./6.25.3/postgres.symbols.ignore)
+
+2. Add ignored types to `gpdb_src/.abi-check/<base version>/postgres.types.ignore` (one type per line).
+   - `<base version>`: The baseline version of Greenplum. If we want to ensure the ABI isn't broken between the `6.25.3` release and the latest `6X_STABLE`. The baseline version of Greenplum is `6.25.3`. See: [./6.25.3/postgres.types.ignore](./6.25.3/postgres.types.ignore)
+
+3. Pass these two files to `abi-compliance-checker` and it will produce a report in HTML format.
+   ```
+   abi-compliance-checker -skip-symbols gpdb_src/.abi-check/<base version>/postgres.symbols.ignore \
+                          -skip-types   gpdb_src/.abi-check/<base version>/postgres.types.ignore \
+                          -lib postgres \
+                          -old greenplum-<base version>.dump
+                          -new greenplum-new.dump
+   ```
+   It will produce a ABI report in `./compat_reports/postgres/X_to_Y/compat_report.html`.
+
+## View the ABI compatibility report
+
+### View the report locally
+
+You can either open the HTML report in your browser or dump it to stdout using `lynx -dump compat_reports/postgres/X_to_Y/compat_report.html`.
+
+## View the report from GitHub Action
+
+1. Navigate to the "Summary" page of the test.
+2. Click the report and download it.
+3. View the report as above.
+
+![./.images/download-report-from-gh-action.png](./.images/download-report-from-gh-action.png)
diff --git a/.github/workflows/greenplum-abi-tests.yml b/.github/workflows/greenplum-abi-tests.yml
@@ -0,0 +1,178 @@
+name: Greenplum ABI Tests
+
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - 'concourse/scripts/**'
+      - 'src/**'
+      - '.github/workflows/**'
+      - '.github/scripts/**'
+      - '.abi-check/**'
+
+  push:
+    branches:
+      - 6X_STABLE
+    paths:
+      - 'concourse/scripts/**'
+      - 'src/**'
+      - '.github/workflows/**'
+      - '.github/scripts/**'
+      - '.abi-check/**'
+
+jobs:
+  abi-dump-setup:
+    runs-on: ubuntu-latest
+    outputs:
+      BASELINE_REF: ${{ steps.vars.outputs.BASELINE_REF }}
+      BASELINE_VERSION: ${{ steps.vars.outputs.BASELINE_VERSION }}
+      ABI_LIBS: ${{ steps.vars.outputs.ABI_LIBS }}
+      ABI_HEADERS: ${{ steps.vars.outputs.ABI_HEADERS }}
+    steps:
+      - name: Fetch source
+        uses: actions/checkout@v3
+
+      - name: Get Greenplum version variables
+        id: vars
+        run: |
+          remote_repo='https://github.com/greenplum-db/gpdb.git'
+          git ls-remote --tags --refs --sort='v:refname' $remote_repo '6.*' | tail -n 1  > baseline_version_ref
+          baseline_ref=$(cat baseline_version_ref | awk '{print $1}')
+          baseline_version=$(cat baseline_version_ref | awk '{print $2}')
+          echo "BASELINE_REF=${baseline_ref}" | tee -a $GITHUB_OUTPUT
+          echo "BASELINE_VERSION=${baseline_version#'refs/tags/'}" | tee -a $GITHUB_OUTPUT
+          echo "ABI_LIBS=postgres" | tee -a $GITHUB_OUTPUT
+          echo "ABI_HEADERS=." | tee -a $GITHUB_OUTPUT
+
+      - name: Upload symbol/type checking exception list
+        uses: actions/upload-artifact@v3
+        with:
+          name: exception_lists
+          path: '.abi-check/${{ steps.vars.outputs.BASELINE_VERSION }}/'
+
+  abi-dump:
+    needs: abi-dump-setup
+    runs-on: ubuntu-latest
+    container: gcr.io/data-gpdb-public-images/gpdb6-rocky8-build
+    strategy:
+      matrix:
+        name:
+          - build-baseline
+          - build-latest
+        include:
+          - name: build-baseline
+            repo: greenplum-db/gpdb
+            ref: ${{ needs.abi-dump-setup.outputs.BASELINE_VERSION }}
+          - name: build-latest
+            repo: ${{ github.repository }}
+            ref: ${{ github.sha }}
+
+    steps:
+      ## FIXME: abi-dumper requires 'Universal Ctags' but the package manager only provides
+      ## 'Exuberant Ctags'.
+      - name: Install universal-ctags.
+        run: |
+          wget 'https://github.com/universal-ctags/ctags-nightly-build/releases/download/2023.07.05%2Bafdae39c0c2e508d113cbc570f4635b96159840c/uctags-2023.07.05-linux-x86_64.tar.xz'
+          tar -xf uctags-2023.07.05-linux-x86_64.tar.xz
+          cp uctags-2023.07.05-linux-x86_64/bin/* /usr/bin/
+          which ctags
+
+      - name: Download Greenplum source code
+        uses: actions/checkout@v3
+        with:
+          repository: ${{ matrix.repo }}
+          ref: ${{ matrix.ref }}
+          submodules: recursive
+          fetch-depth: 0 # Specify '0' to fetch all history for all branches and tags.
+          path: gpdb_src
+
+      - name: Install abi-dumper
+        run: |
+          yum install -y epel-release
+          yum install -y abi-dumper
+
+      - name: Build Greenplum
+        run: |
+          ## TODO: Since abi-dumper requires debug info and it's hard to inject CFLAGS via the script for
+          ## releasing Greenplum, we have to manually configure it here. Probably we can improve it in future.
+          export PATH=/opt/python-3.9.13/bin:/opt/python-2.7.18/bin:$PATH
+          pushd gpdb_src
+          CC='gcc -m64' \
+          CFLAGS='-Og -g3 -Wno-maybe-uninitialized' LDFLAGS='-Wl,--enable-new-dtags -Wl,--export-dynamic' \
+          ./configure --with-quicklz --disable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy \
+                      --enable-orca --with-libxml --with-pythonsrc-ext --with-uuid=e2fs --with-pgport=5432 --enable-tap-tests \
+                      --enable-debug-extensions --with-perl --with-python --with-openssl --with-pam --with-ldap --with-includes="" \
+                      --with-libraries="" --disable-rpath \
+                      --prefix=/usr/local/greenplum-db-devel \
+                      --mandir=/usr/local/greenplum-db-devel/man
+          make -j`nproc` && make install
+
+      - name: Dump ABI
+        run: |
+          abi-dumper -lver ${{ matrix.ref }} -skip-cxx -public-headers /usr/local/greenplum-db-devel/include/${{ needs.abi-dump-setup.outputs.ABI_HEADERS }} -o postgres-${{ matrix.ref }}.abi /usr/local/greenplum-db-devel/bin/postgres
+
+      - name: Upload ABI files
+        uses: actions/upload-artifact@v3
+        with:
+          name: ${{ matrix.name }}
+          path: '*${{ matrix.ref }}.abi'
+
+  abi-compare:
+    needs:
+      - abi-dump-setup
+      - abi-dump
+    runs-on: ubuntu-latest
+    container: gcr.io/data-gpdb-public-images/gpdb6-rocky8-build
+    steps:
+      - name: Download baseline
+        uses: actions/download-artifact@v3
+        with:
+          name: build-baseline
+          path: build-baseline/
+      - name: Download latest
+        uses: actions/download-artifact@v3
+        with:
+          name: build-latest
+          path: build-latest/
+
+      - name: Download exception lists
+        uses: actions/download-artifact@v3
+        with:
+          name: exception_lists
+          path: exception_lists/
+
+      - name: Install abi-compliance-checker and report viewer (lynx)
+        run: |
+          yum install -y epel-release
+          yum install -y abi-compliance-checker
+          yum install -y --enablerepo=powertools lynx
+
+      - name: Compare ABI
+        run: |
+          SKIP_POSTGRES_SYMBOLS_LIST="exception_lists/postgres.symbols.ignore"
+          SKIP_POSTGRES_SYMBOLS_OPTION=""
+          if [[ -f "$SKIP_POSTGRES_SYMBOLS_LIST" ]]; then
+            SKIP_POSTGRES_SYMBOLS_OPTION="-skip-symbols ${SKIP_POSTGRES_SYMBOLS_LIST}"
+          fi
+          SKIP_POSTGRES_TYPES_LIST="exception_lists/postgres.types.ignore"
+          SKIP_POSTGRES_TYPES_OPTION=""
+          if [[ -f "$SKIP_POSTGRES_TYPES_LIST" ]]; then
+            SKIP_POSTGRES_TYPES_OPTION="-skip-types ${SKIP_POSTGRES_TYPES_LIST}"
+          fi
+          abi-compliance-checker ${SKIP_POSTGRES_SYMBOLS_OPTION} \
+                                 ${SKIP_POSTGRES_TYPES_OPTION} \
+                                 -lib postgres \
+                                 -old build-baseline/postgres*.abi \
+                                 -new build-latest/postgres*.abi
+
+      - name: Print out ABI report
+        if: always()
+        run: |
+          lynx -dump $(find compat_reports/ | grep html)
+
+      - name: Upload ABI Comparison
+        if: always()
+        uses: actions/upload-artifact@v3
+        with:
+          name: compat-report-${{ github.sha }}
+          path: compat_reports/
diff --git a/concourse/pipelines/gpdb_6X_STABLE-generated.yml b/concourse/pipelines/gpdb_6X_STABLE-generated.yml
@@ -12,7 +12,7 @@
 ## file (example: templates/gpdb-tpl.yml) and regenerate the pipeline
 ## using appropriate tool (example: gen_pipeline.py -t prod).
 ## ----------------------------------------------------------------------
-## Generated by gen_pipeline.py at: 2023-09-06 12:50:28.297086
+## Generated by gen_pipeline.py at: 2023-10-31 10:46:55.243925
 ## Template file: gpdb-tpl.yml
 ## OS Types: 
 ## Test Sections: ['icw', 'cli', 'aa', 'release']
@@ -1348,7 +1348,7 @@ jobs:
         <<: *ccp_default_params
         vars:
           <<: *ccp_default_vars
-          instance_type: n1-standard-4
+          instance_type: n1-highmem-4
           number_of_nodes: 2
     - task: gen_cluster
       file: ccp_src/ci/tasks/gen_cluster.yml
@@ -1436,7 +1436,6 @@ jobs:
         - unit_tests_gporca_rocky8
         - gpdb_pitr_rocky8
         - interconnect_rocky8
-        - icw_extensions_gpcloud_rocky8
         - gpexpand_rocky8
         - pg_upgrade_rocky8
       - get: gpdb_src
@@ -1454,7 +1453,6 @@ jobs:
         - unit_tests_gporca_rocky8
         - gpdb_pitr_rocky8
         - interconnect_rocky8
-        - icw_extensions_gpcloud_rocky8
         - gpexpand_rocky8
         - pg_upgrade_rocky8
         trigger: true

diff --git a/concourse/pipelines/templates/gpdb-tpl.yml b/concourse/pipelines/templates/gpdb-tpl.yml
@@ -849,8 +849,8 @@ resources:
   type: time
   source:
     location: America/Los_Angeles
-    start: ((reduced-frequency-trigger-start-[[ os_type ]]))
-    stop: ((reduced-frequency-trigger-stop-[[ os_type ]]))
+    start: (("reduced-frequency-trigger-start-[[ os_type ]]"))
+    stop: (("reduced-frequency-trigger-stop-[[ os_type ]]"))
     {% if os_type != "centos7" %}
     days: [Monday]
     {% else %}
@@ -1718,7 +1718,7 @@ jobs:
         <<: *ccp_default_params
         vars:
           <<: *ccp_default_vars
-          instance_type: n1-standard-4
+          instance_type: n1-highmem-4
           number_of_nodes: 2
     - task: gen_cluster
       file: ccp_src/ci/tasks/gen_cluster.yml
@@ -1811,7 +1811,6 @@ jobs:
         - unit_tests_gporca_[[ os_type ]]
         - gpdb_pitr_[[ os_type ]]
         - interconnect_[[ os_type ]]
-        - icw_extensions_gpcloud_[[ os_type ]]
         - gpexpand_[[ os_type ]]
         - pg_upgrade_[[ os_type ]]
       - get: gpdb_src
@@ -1835,7 +1834,6 @@ jobs:
         - unit_tests_gporca_[[ os_type ]]
         - gpdb_pitr_[[ os_type ]]
         - interconnect_[[ os_type ]]
-        - icw_extensions_gpcloud_[[ os_type ]]
         - gpexpand_[[ os_type ]]
         - pg_upgrade_[[ os_type ]]
         trigger: true

diff --git a/concourse/scripts/verify_gpdb_versions.bash b/concourse/scripts/verify_gpdb_versions.bash
@@ -20,8 +20,6 @@ assert_postgres_version_matches() {
   fi
 }
 
-yum -d0 -y install git
-
 GREENPLUM_INSTALL_DIR=/usr/local/greenplum-db-devel
 GPDB_SRC_SHA=$(cd gpdb_src && git rev-parse HEAD)
 

diff --git a/concourse/tasks/verify_gpdb_versions.yml b/concourse/tasks/verify_gpdb_versions.yml
@@ -4,8 +4,8 @@ platform: linux
 image_resource:
   type: registry-image
   source:
-    repository: centos
-    tag: 7
+    repository: gcr.io/data-gpdb-public-images/gpdb6-rocky8-build
+    tag: latest
 
 inputs:
   - name: gpdb_src

diff --git a/gpAux/Makefile b/gpAux/Makefile
@@ -604,8 +604,8 @@ copylibs :
 	    echo "INFO: Python not found on this platform, $(BLD_ARCH), not copying it into the GPDB package."; \
 	fi
 	# Create the python3.9 directory to flag to build scripts that python has been handled
-	mkdir -p $(INSTLOC)/ext/python3.9
 	@if [ ! -z "$(PYTHONHOME39)" ]; then \
+	    mkdir -p $(INSTLOC)/ext/python3.9; \
 	    echo "Copying python3.9, ., from $(PYTHONHOME39) into $(INSTLOC)/ext/python3.9..."; \
 	    (cd $(PYTHONHOME39) && tar cf - .) | (cd $(INSTLOC)/ext/python3.9/ && tar xpf -); \
 		echo "...DONE"; \

diff --git a/gpAux/extensions/pgbouncer/source b/gpAux/extensions/pgbouncer/source
diff --git a/gpMgmt/bin/Makefile b/gpMgmt/bin/Makefile
@@ -7,7 +7,7 @@ ifneq "$(wildcard $(top_builddir)/src/Makefile.global)" ""
 include $(top_builddir)/src/Makefile.global
 endif
 
-SUBDIRS = stream gpcheckcat_modules gpconfig_modules gpssh_modules gppylib lib
+SUBDIRS = stream gpcheckcat_modules gpconfig_modules gpssh_modules gppylib lib el8_migrate_locale
 SUBDIRS += ifaddrs
 
 $(recurse)