forked from greenplum-db/gpdb-archive
-
Notifications
You must be signed in to change notification settings - Fork 22
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'adb-6.x-dev' into ADBDEV-1938-optimize-2
- Loading branch information
Showing
222 changed files
with
8,947 additions
and
997 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| DummySymbol | ||
| ConfigureNamesInt_gp |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| DummyType |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # Check the compatibility of Greenplum ABI. | ||
|
|
||
| ## Introduction | ||
|
|
||
| We use the [`abi-dumper`](https://github.com/lvc/abi-dumper) and [`abi-compliance-checker`](https://github.com/lvc/abi-compliance-checker/) to check the Greenplum's ABI. We also use the [GitHub action](../.github/workflows/greenplum-abi-tests.yml) to automate this job. | ||
|
|
||
| ## Requirements | ||
|
|
||
| `abi-dumper` requires the binary being compiled with `-Og -g3`, hence the `CFLAGS` for configuration looks like: | ||
|
|
||
| ```bash | ||
| ## GCC's maybe-uninitialized checker may produce false positives with different | ||
| ## levels of optimizations. To prevent building failures, we append the '-Wno-maybe-uninitialized' | ||
| ## to the $CFLAGS as well. | ||
| CFLAGS='-Og -g3 -Wno-maybe-uninitialized' ./configure --with-xxx --with-yyy --with-zzz | ||
| ``` | ||
|
|
||
| ## Check the ABI's compatibility | ||
|
|
||
| Several binaries are shipped in Greenplum, e.g., `$GPHOME/bin/postgres`, `$GPHOME/lib/libpq.so`, etc. Since the `postgres` binary are referenced by many extensions, the ABI compatibility of it is the most important. The following steps illustrate how to check the ABI compatibility of the `postgres` binary. | ||
|
|
||
| 1. Dump the ABI information of one `postgres` binary. | ||
| ``` | ||
| abi-dumper $GPHOME/bin/postgres -lver <version of the binary> -o <output file> | ||
| ``` | ||
| - `<version of the binary>`: The version of the binary. You can give it some reasonable name, e.g., `6.25.3` to indicate the binary is built from '6.25.3' tag. | ||
| - `<output file>`: The file path for dumping the ABI information, e.g., `greenplum-6.25.3.dump` | ||
|
|
||
| 2. Dump the ABI information of another `postgres` binary (same as the step 1). | ||
|
|
||
| 3. Compare the ABI between these two binaries with `abi-compliance-checker`. | ||
| ``` | ||
| abi-compliance-checker \ | ||
| -lib <library name> \ | ||
| -old <output file from step1> \ | ||
| -new <output file from step2> | ||
| ``` | ||
| - `<library name>`: The name of the library, e.g., `postgres`. | ||
|
|
||
| 4. By default, the `abi-compliance-checker` will produce an HTML web page and there will be detailed information about ABI changes. | ||
|
|
||
| ## Ignore the "Safe ABI breaking change" | ||
|
|
||
| There might be "safe ABI breaking changes", e.g., some symbol being removed and not referenced by any extensions or programs. Here are steps on how to suppress such errors. | ||
|
|
||
| 1. Add ignored symbols to `gpdb_src/.abi-check/<base version>/postgres.symbols.ignore` (one symbol per line). | ||
| - `<base version>`: The baseline version of Greenplum. If we want to ensure the ABI isn't broken between the `6.25.3` release and the latest `6X_STABLE`. The baseline version of Greenplum is `6.25.3`. See: [./6.25.3/postgres.symbols.ignore](./6.25.3/postgres.symbols.ignore) | ||
|
|
||
| 2. Add ignored types to `gpdb_src/.abi-check/<base version>/postgres.types.ignore` (one type per line). | ||
| - `<base version>`: The baseline version of Greenplum. If we want to ensure the ABI isn't broken between the `6.25.3` release and the latest `6X_STABLE`. The baseline version of Greenplum is `6.25.3`. See: [./6.25.3/postgres.types.ignore](./6.25.3/postgres.types.ignore) | ||
|
|
||
| 3. Pass these two files to `abi-compliance-checker` and it will produce a report in HTML format. | ||
| ``` | ||
| abi-compliance-checker -skip-symbols gpdb_src/.abi-check/<base version>/postgres.symbols.ignore \ | ||
| -skip-types gpdb_src/.abi-check/<base version>/postgres.types.ignore \ | ||
| -lib postgres \ | ||
| -old greenplum-<base version>.dump | ||
| -new greenplum-new.dump | ||
| ``` | ||
| It will produce a ABI report in `./compat_reports/postgres/X_to_Y/compat_report.html`. | ||
|
|
||
| ## View the ABI compatibility report | ||
|
|
||
| ### View the report locally | ||
|
|
||
| You can either open the HTML report in your browser or dump it to stdout using `lynx -dump compat_reports/postgres/X_to_Y/compat_report.html`. | ||
|
|
||
| ## View the report from GitHub Action | ||
|
|
||
| 1. Navigate to the "Summary" page of the test. | ||
| 2. Click the report and download it. | ||
| 3. View the report as above. | ||
|
|
||
|  |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| name: Greenplum ABI Tests | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| pull_request: | ||
| paths: | ||
| - 'concourse/scripts/**' | ||
| - 'src/**' | ||
| - '.github/workflows/**' | ||
| - '.github/scripts/**' | ||
| - '.abi-check/**' | ||
|
|
||
| push: | ||
| branches: | ||
| - 6X_STABLE | ||
| paths: | ||
| - 'concourse/scripts/**' | ||
| - 'src/**' | ||
| - '.github/workflows/**' | ||
| - '.github/scripts/**' | ||
| - '.abi-check/**' | ||
|
|
||
| jobs: | ||
| abi-dump-setup: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| BASELINE_REF: ${{ steps.vars.outputs.BASELINE_REF }} | ||
| BASELINE_VERSION: ${{ steps.vars.outputs.BASELINE_VERSION }} | ||
| ABI_LIBS: ${{ steps.vars.outputs.ABI_LIBS }} | ||
| ABI_HEADERS: ${{ steps.vars.outputs.ABI_HEADERS }} | ||
| steps: | ||
| - name: Fetch source | ||
| uses: actions/checkout@v3 | ||
|
|
||
| - name: Get Greenplum version variables | ||
| id: vars | ||
| run: | | ||
| remote_repo='https://github.com/greenplum-db/gpdb.git' | ||
| git ls-remote --tags --refs --sort='v:refname' $remote_repo '6.*' | tail -n 1 > baseline_version_ref | ||
| baseline_ref=$(cat baseline_version_ref | awk '{print $1}') | ||
| baseline_version=$(cat baseline_version_ref | awk '{print $2}') | ||
| echo "BASELINE_REF=${baseline_ref}" | tee -a $GITHUB_OUTPUT | ||
| echo "BASELINE_VERSION=${baseline_version#'refs/tags/'}" | tee -a $GITHUB_OUTPUT | ||
| echo "ABI_LIBS=postgres" | tee -a $GITHUB_OUTPUT | ||
| echo "ABI_HEADERS=." | tee -a $GITHUB_OUTPUT | ||
| - name: Upload symbol/type checking exception list | ||
| uses: actions/upload-artifact@v3 | ||
| with: | ||
| name: exception_lists | ||
| path: '.abi-check/${{ steps.vars.outputs.BASELINE_VERSION }}/' | ||
|
|
||
| abi-dump: | ||
| needs: abi-dump-setup | ||
| runs-on: ubuntu-latest | ||
| container: gcr.io/data-gpdb-public-images/gpdb6-rocky8-build | ||
| strategy: | ||
| matrix: | ||
| name: | ||
| - build-baseline | ||
| - build-latest | ||
| include: | ||
| - name: build-baseline | ||
| repo: greenplum-db/gpdb | ||
| ref: ${{ needs.abi-dump-setup.outputs.BASELINE_VERSION }} | ||
| - name: build-latest | ||
| repo: ${{ github.repository }} | ||
| ref: ${{ github.sha }} | ||
|
|
||
| steps: | ||
| ## FIXME: abi-dumper requires 'Universal Ctags' but the package manager only provides | ||
| ## 'Exuberant Ctags'. | ||
| - name: Install universal-ctags. | ||
| run: | | ||
| wget 'https://github.com/universal-ctags/ctags-nightly-build/releases/download/2023.07.05%2Bafdae39c0c2e508d113cbc570f4635b96159840c/uctags-2023.07.05-linux-x86_64.tar.xz' | ||
| tar -xf uctags-2023.07.05-linux-x86_64.tar.xz | ||
| cp uctags-2023.07.05-linux-x86_64/bin/* /usr/bin/ | ||
| which ctags | ||
| - name: Download Greenplum source code | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| repository: ${{ matrix.repo }} | ||
| ref: ${{ matrix.ref }} | ||
| submodules: recursive | ||
| fetch-depth: 0 # Specify '0' to fetch all history for all branches and tags. | ||
| path: gpdb_src | ||
|
|
||
| - name: Install abi-dumper | ||
| run: | | ||
| yum install -y epel-release | ||
| yum install -y abi-dumper | ||
| - name: Build Greenplum | ||
| run: | | ||
| ## TODO: Since abi-dumper requires debug info and it's hard to inject CFLAGS via the script for | ||
| ## releasing Greenplum, we have to manually configure it here. Probably we can improve it in future. | ||
| export PATH=/opt/python-3.9.13/bin:/opt/python-2.7.18/bin:$PATH | ||
| pushd gpdb_src | ||
| CC='gcc -m64' \ | ||
| CFLAGS='-Og -g3 -Wno-maybe-uninitialized' LDFLAGS='-Wl,--enable-new-dtags -Wl,--export-dynamic' \ | ||
| ./configure --with-quicklz --disable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy \ | ||
| --enable-orca --with-libxml --with-pythonsrc-ext --with-uuid=e2fs --with-pgport=5432 --enable-tap-tests \ | ||
| --enable-debug-extensions --with-perl --with-python --with-openssl --with-pam --with-ldap --with-includes="" \ | ||
| --with-libraries="" --disable-rpath \ | ||
| --prefix=/usr/local/greenplum-db-devel \ | ||
| --mandir=/usr/local/greenplum-db-devel/man | ||
| make -j`nproc` && make install | ||
| - name: Dump ABI | ||
| run: | | ||
| abi-dumper -lver ${{ matrix.ref }} -skip-cxx -public-headers /usr/local/greenplum-db-devel/include/${{ needs.abi-dump-setup.outputs.ABI_HEADERS }} -o postgres-${{ matrix.ref }}.abi /usr/local/greenplum-db-devel/bin/postgres | ||
| - name: Upload ABI files | ||
| uses: actions/upload-artifact@v3 | ||
| with: | ||
| name: ${{ matrix.name }} | ||
| path: '*${{ matrix.ref }}.abi' | ||
|
|
||
| abi-compare: | ||
| needs: | ||
| - abi-dump-setup | ||
| - abi-dump | ||
| runs-on: ubuntu-latest | ||
| container: gcr.io/data-gpdb-public-images/gpdb6-rocky8-build | ||
| steps: | ||
| - name: Download baseline | ||
| uses: actions/download-artifact@v3 | ||
| with: | ||
| name: build-baseline | ||
| path: build-baseline/ | ||
| - name: Download latest | ||
| uses: actions/download-artifact@v3 | ||
| with: | ||
| name: build-latest | ||
| path: build-latest/ | ||
|
|
||
| - name: Download exception lists | ||
| uses: actions/download-artifact@v3 | ||
| with: | ||
| name: exception_lists | ||
| path: exception_lists/ | ||
|
|
||
| - name: Install abi-compliance-checker and report viewer (lynx) | ||
| run: | | ||
| yum install -y epel-release | ||
| yum install -y abi-compliance-checker | ||
| yum install -y --enablerepo=powertools lynx | ||
| - name: Compare ABI | ||
| run: | | ||
| SKIP_POSTGRES_SYMBOLS_LIST="exception_lists/postgres.symbols.ignore" | ||
| SKIP_POSTGRES_SYMBOLS_OPTION="" | ||
| if [[ -f "$SKIP_POSTGRES_SYMBOLS_LIST" ]]; then | ||
| SKIP_POSTGRES_SYMBOLS_OPTION="-skip-symbols ${SKIP_POSTGRES_SYMBOLS_LIST}" | ||
| fi | ||
| SKIP_POSTGRES_TYPES_LIST="exception_lists/postgres.types.ignore" | ||
| SKIP_POSTGRES_TYPES_OPTION="" | ||
| if [[ -f "$SKIP_POSTGRES_TYPES_LIST" ]]; then | ||
| SKIP_POSTGRES_TYPES_OPTION="-skip-types ${SKIP_POSTGRES_TYPES_LIST}" | ||
| fi | ||
| abi-compliance-checker ${SKIP_POSTGRES_SYMBOLS_OPTION} \ | ||
| ${SKIP_POSTGRES_TYPES_OPTION} \ | ||
| -lib postgres \ | ||
| -old build-baseline/postgres*.abi \ | ||
| -new build-latest/postgres*.abi | ||
| - name: Print out ABI report | ||
| if: always() | ||
| run: | | ||
| lynx -dump $(find compat_reports/ | grep html) | ||
| - name: Upload ABI Comparison | ||
| if: always() | ||
| uses: actions/upload-artifact@v3 | ||
| with: | ||
| name: compat-report-${{ github.sha }} | ||
| path: compat_reports/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule source
updated
from 331c06 to cbbdde
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.