Skip to content

Commit

Permalink
ingest benchmark: after effective_io_concurrency = 100 we can increas…
Browse files Browse the repository at this point in the history
…e compute side parallelism (#9904)

## Problem

ingest benchmark tests project migration to Neon involving steps
- COPY relation data
- create indexes
- create constraints

Previously we used only 4 copy jobs, 4 create index jobs and 7
maintenance workers. After increasing effective_io_concurrency on
compute we see that we can sustain more parallelism in the ingest bench

## Summary of changes

Increase copy jobs to 8, create index jobs to 8 and maintenance workers
to 16
  • Loading branch information
Bodobolero authored Nov 27, 2024
1 parent 2b788cb commit 277c33b
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions test_runner/performance/test_perf_ingest_using_pgcopydb.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,13 @@ def build_pgcopydb_command(pgcopydb_filter_file: Path, test_output_dir: Path):
"--no-acl",
"--skip-db-properties",
"--table-jobs",
"4",
"8",
"--index-jobs",
"4",
"8",
"--restore-jobs",
"4",
"8",
"--split-tables-larger-than",
"10GB",
"5GB",
"--skip-extensions",
"--use-copy-binary",
"--filters",
Expand Down Expand Up @@ -136,7 +136,7 @@ def run_command_and_log_output(command, log_file_path: Path):
"LD_LIBRARY_PATH": f"{os.getenv('PGCOPYDB_LIB_PATH')}:{os.getenv('PG_16_LIB_PATH')}",
"PGCOPYDB_SOURCE_PGURI": cast(str, os.getenv("BENCHMARK_INGEST_SOURCE_CONNSTR")),
"PGCOPYDB_TARGET_PGURI": cast(str, os.getenv("BENCHMARK_INGEST_TARGET_CONNSTR")),
"PGOPTIONS": "-c maintenance_work_mem=8388608 -c max_parallel_maintenance_workers=7",
"PGOPTIONS": "-c maintenance_work_mem=8388608 -c max_parallel_maintenance_workers=16",
}
# Combine the current environment with custom variables
env = os.environ.copy()
Expand Down

1 comment on commit 277c33b

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6941 tests run: 6631 passed, 2 failed, 308 skipped (full report)


Failures on Postgres 17

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_compute_catalog[debug-pg17] or test_idle_checkpoints[debug-pg17]"
Flaky tests (2)

Postgres 17

Postgres 16

Test coverage report is not available

The comment gets automatically updated with the latest test results
277c33b at 2024-11-27T11:12:11.403Z :recycle:

Please sign in to comment.