Sync 2.3.0 changes #26
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Note: `TextDatumGetCString()` returns a null-terminated CString.
```
[ 12%] Building C object CMakeFiles/diskquota.dir/src/diskquota.c.o
In file included from /home/v/workspace/diskquota/src/diskquota.h:16,
from /home/v/workspace/diskquota/src/diskquota.c:18:
/home/v/workspace/diskquota/src/diskquota.c: In function ‘diskquota_status_schema_version’:
/home/v/.local/gpdb7/include/postgresql/server/c.h:957:25: warning: ‘strncpy’ specified bound 64 equals destinatio
n size [-Wstringop-truncation]
957 | strncpy(_dst, (src), _len); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/v/workspace/diskquota/src/diskquota.c:1548:9: note: in expansion of macro ‘StrNCpy’
1548 | StrNCpy(version, vv, sizeof(version));
| ^~~~~~~
```
Co-authored-by: Hao Zhang <[email protected]>
Fix bug produced by #339 Co-authored-by: Xing Guo [email protected]
fix pipeline resource
# Problem Recording active tables and permission checking on segments rely on `monitored_dbid_cache`. But after mirror switching, the data in shared memory is lost, and the above behaviors will be blocked. # Solution Segments update their `monitored_dbid_cache` after receiving pull_active_tables_oid requests every naptime.
To achieve upgrading directly from 2.0 to 2.2, we should do the following things: - cherry-pick this PR in diskquota-2.0, diskquota-2.1 and diskquota-2.2. - set the shared_preload_libraries as `diskquota-2.2`: `gpconfig -c shared_preload_libraries -v 'diskquota-2.2'` - restart cluster: `gpstop -ar` - execute the following SQLs: ``` ALTER extension diskquota update to '2.2'; ```
Get the latest package version for release resource.
- Because of 97f1f9b, the sql file directory has been changed. The sql version check wouldn't work since it cannot find the sql files anymore. Change it to the correct ddl directory. - 'exec_program' is deprecated, use 'execute_process' instead. - 'git tag | sort' returns the latest version among all branches, but not the closest tag to the current commit. Use 'git describe --tags' instead. So the upgrade version check will work for the 2.1.x patch release.
After creating extension, the `diskquota.state` is always `unready` due to the change greenplum-db/gpdb#15239. It makes the test timeout. We disable the job of gp7 in release/pr/merge pipeline until the problem is fixed.
- By #340, diskquota should be able to upgraded directly from any previous version. A script is added to test this. - Modify the cmakefile so before installing/packaing, only previous so files will copied. This would help us to make patch release for 2.0/2.1. --------- Co-authored-by: Zhang Hao <[email protected]>
Co-authored-by: Chen Mulong [email protected]
- bump version to 2.2.2 - Modify the check procedure of alter extension.
Fix from #346
git-subtree-dir: concourse/lib git-subtree-split: d51adf5d59e4e5caefb678b29c4335881040c7d2
Migrage changes pivotal/pgaudit#13 here
Change the CI pipeline to release diskquota for GP7. TODO: - enable regress test on CI. - enable activatestandby test for GP7 on CI. - fix regress test for GP7. The view in GP7 will be treated as a relation by diskquota. After `CREATE EXTENSION`, the test case should execute `SELECT diskquota.init_table_size_table()` to make the diskquota.state ready on GP7.
Fix the following issues:
diskquota cannot correctly print a warning when calling
`create extension` in a non-empty database. The diskquota.state shows
whether the current database is empty. Previously, we updated the
status to diskquota.state during `create extension` and queried the
status in UDF `diskquota.diskquota_start_worker`. When querying the
database status, the relations whose `relkind in ('v', 'c', 'f')` are
skipped, while all relations are filtered when updating the database
status. This patch merges the two SQL statements to `INSERT RETURNING`
to solve this problem.
Benefit: we won't need to upgrade the minor version of diskquota when
changing the statement.
Remove useless test `test_update_db_cache`. The behavior of the
bgworker scheduler is changed, so this test is no longer helpful.
Update the upgrade test script `2.2_set_quota`. Since the above issue,
there should be `select diskquota.init_table_size_table()` in
`2.2_set_quota`.
Add upgrade test for gp7.
Fix upgrade test for gp6.
Fix the standby test on GP7.
All tables in diskquota use primary keys. Unique index tables are created for primary keys. Unique index tables can not be created for tables with the option appendonly=true. This issue only occurs in gp6 since `set gp_default_storage_options='appendonly=true';` is disabled in gp7. Solution: add `appendonly=false` to each table with the primary key. Co-authored-by: Evgeniy Ratkov <[email protected]>
During the upgrade procedure, when diskquota.so is not in shared_preload_libraries, check whether the current version is the final version: - If yes, throw an error to the user. - Otherwise, return from _PG_init(). Co-authored-by: Sasasu <[email protected]>
This GUC shows the max number of database diskquota can monitor.
- Fix the compilation issue caused by GP7. `libpq.so` needs to be linked. See greenplum-db/gpdb#16234. - Add GP7 jobs back to the pipeline.
`diskquota.max_quotas` shows the max number of quotas on the cluster.
In the current version of diskquota, quotas means:
- role
- schema
- role_tablespace
- schema_tablespace
The number of quota =
role + schema + (role * tablespace) + (schema * tablespace).
- skip add last version to the current build. - skip upgrade test.
Co-authored-by: Chen Mulong <[email protected]> Co-authored-by: Xing Guo <[email protected]>
Merge all types of quota info maps into one in each bgworker.
The package name is incorrect when releasing package in Rocky9/RHEL9 platforms (<package>-<version>-unknown_x86_64.tar.gz). This patch fix it to <package>-<version>-rhel9_x86_64.tar.gz. Cherry-pick pivotal/timestamp9#41
This reverts commit e6899aa.
Diskquota calculates sizes and stores information in the diskquota.table_size table periodically with a pause in diskquota.naptime, 2 seconds by default. If we restart the cluster during this pause, then diskquota will lose all changes that have occurred since the last save to the diskquota.table_size table. We could create temporary tables, wait when it will be flushed to diskquota.table_size table, restart the cluster, and diskquota would remember the information about the temporary tables. Or we could delete the tables, restart the cluster, and again diskquota will remember information about the deleted tables. This happens because at the start of the cluster, diskquota remembers all the information written to the diskquota.table_size table, but does not check that some tables may have already been deleted. As a solution, we invalidate diskquota.table_size during diskquota worker start in addition to pg_class validation.
Removed useless commands from `concourse/scripts/entry.sh`, because our docker images already have source of GPDB at `/home/gpadmin/gpdb_src` and do not need to create or configure this directory. Added file `arenadata/README.md` with information about how to run tests at docker container. Removed `--load-extension=diskquota_test` from `CMakeLists.txt` at regression tests. Creating extension `diskquota_test` requires configured extension `diskquota`. But `diskquota` is configured when `tests/regress/sql/config.sql` is executed, and load-extension is done before this test is started. At all tests (where `diskquota_test` is used) command `CREATE EXTENSION diskquota_test;` is executed. This commit is cherry-picked from commit d5b5d1e. Original commit was modified because at commit fba3d06 the most of scripts (which are needed for runnig tests) were move to private upstream repo. Script entry.sh was restored from our last version and moved to concourse directory. I added variable OS_NAME="rhel9" at running "build.sh" and "test.sh" in entry.sh to disable upgrade tests. Path "/home/gpadmin/bin_diskquota" was changed to "/home/gpadmin/diskquota_artifacts" not to do changes at our CI
|
May be move last big comment to PR description? |
RekGRpth
approved these changes
Nov 29, 2023
bimboterminator1
approved these changes
Nov 29, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our commits, which were reverted because of conflicts:
commit 3b06e37
commit 50ed2e4
commit d5b5d1e
commit e6899aa
Commits, which were cherry-picked with changes
commit e6899aa
commit d5b5d1e
Commit which is not needed:
commit 50ed2e4
Commit, which was not cherry-picked because regression test fails:
commit 3b06e37
test fails at
test_create_extension:At log I found that bgworker process was stopped:
Without this commit bgworker is stared after each
CREATE EXTENSION diskquotacommands: