Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage & Compute release 2024-10-14 #9372

Merged
merged 59 commits into from
Oct 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
47c3c9a
Fix update of statistic for LFC/prefetch (#9272)
knizhnik Oct 7, 2024
8cd7b5b
proxy: rename console -> control_plane, rename web -> console_redirec…
conradludgate Oct 7, 2024
ad267d8
proxy: Move module base files into module directory (#9297)
cloneable Oct 7, 2024
323bd01
Make sure BufferTag padding bytes are cleared in hash keys (#9292)
hlinnaka Oct 7, 2024
2272dc8
feat(compute_tools): Create JWKS Postgres roles without attributes (#…
devjv Oct 7, 2024
99d4c18
Replace BUFFERTAGS_EQUAL compatibility macro with new-style function …
hlinnaka Oct 7, 2024
6eba29c
Improve logging on changes in a compute's status
tristan957 Oct 7, 2024
912d47e
storage_broker: update hyper and tonic again (#9299)
arpad-m Oct 7, 2024
cc29def
vm-monitor: Ignore LFC in postgres cgroup memory threshold (#8668)
sharnoff Oct 7, 2024
dbbe57a
Remove unused local vars and a prototype for non-existent function
hlinnaka Oct 7, 2024
11dc5fe
Remove unused static function
hlinnaka Oct 7, 2024
776df96
Fix function prototypes
hlinnaka Oct 7, 2024
129d448
Move "/* fallthrough */" comments so that GCC recognizes them
hlinnaka Oct 7, 2024
e763256
Fix warnings about missing function prototypes
hlinnaka Oct 7, 2024
0a80dbc
neon_write() function is not used on v17
hlinnaka Oct 7, 2024
e553ca9
Silence warnings about mixed declarations and code
hlinnaka Oct 7, 2024
09f2000
Silence warnings about shadowed local variables
hlinnaka Oct 7, 2024
30f7fbc
Add pg_attribute_printf to WalProposerLibLog, per gcc's suggestion
hlinnaka Oct 7, 2024
2ff6d2b
Silence compiler warning about variable only used in assertions
hlinnaka Oct 7, 2024
d7b960c
Silence compiler warning about using variable uninitialized
hlinnaka Oct 7, 2024
17c59ed
Don't override CFLAGS when building neon extension
hlinnaka Oct 7, 2024
18b9715
Remove non-existent entries from .dockerignore (#9209)
hlinnaka Oct 8, 2024
16417d9
Remove get_self_dir()
tristan957 Oct 8, 2024
baf27ba
Fix compiler warnings on macOS (#9319)
bayandin Oct 8, 2024
618680c
storcon: apply all node status changes before handling transitions (#…
VladLazar Oct 8, 2024
5bd8e23
Enable all pyupgrade checks in ruff
tristan957 Oct 8, 2024
e8ae376
Add timeline offload mechanism (#8907)
arpad-m Oct 8, 2024
f87f5a3
tests: Remove redundant log lines when starting an endpoint (#9316)
hlinnaka Oct 9, 2024
211970f
remote_storage: add `DownloadOpts::byte_(start|end)` (#9293)
erikgrinaker Oct 9, 2024
8a138db
tests: Reduce noise from logging renamed files (#9315)
hlinnaka Oct 9, 2024
54d1185
proxy: Unalias hyper1 and replace one use of hyper0 in test (#9324)
cloneable Oct 9, 2024
cc599e2
storcon: make observed state updates more granular (#9276)
VladLazar Oct 9, 2024
fc73971
test_runner: fix path to tpc-h queries (#9327)
bayandin Oct 9, 2024
a181392
safekeeper: add evicted_timelines gauge. (#9318)
arssher Oct 9, 2024
63e7fab
Add /installed_extensions endpoint to collect statistics about extens…
lubennikovaav Oct 9, 2024
bee04b8
pageserver: add direct io config to virtual file (#9214)
yliang412 Oct 9, 2024
eb23d35
tests: Use ThreadedMotoServer python class to launch mock S3 server (…
hlinnaka Oct 9, 2024
72ef0e0
tests: Remove redundant log lines when stopping storage nodes (#9317)
hlinnaka Oct 9, 2024
108a211
added workflow Report Workflow Stats (#9330)
fedordikarev Oct 9, 2024
721803a
Add partial support of extensions for v17: (#9322)
lubennikovaav Oct 9, 2024
7543406
local_proxy: integrate with pg_session_jwt extension (#9086)
conradludgate Oct 9, 2024
878135f
Move PgBenchInitResult.EXTRACTORS to a private module constant
tristan957 Oct 9, 2024
d346458
Improve some typing in test_runner
tristan957 Oct 9, 2024
306094a
add local-proxy suffix to wake-compute requests, respect the returned…
conradludgate Oct 9, 2024
426b1c5
storage controller: use 'infra' JWT scope for node registration (#9343)
jcsp Oct 10, 2024
c2623ff
CODEOWNERS: assign `storage_scrubber` to storage (#9346)
erikgrinaker Oct 10, 2024
9dd80b9
storage_scrubber: fix faulty assertion when no timelines (#9345)
erikgrinaker Oct 10, 2024
264c34d
Move path-related fixtures into their own module (#9304)
tristan957 Oct 10, 2024
07c7143
tests: allow a log warning in test_cli_start_stop_multi (#9320)
jcsp Oct 10, 2024
1f7904c
Enable cargo caching in check-codestyle-rust
tristan957 Oct 10, 2024
006d9df
Add compute_config_dir fixture
tristan957 Oct 10, 2024
53147b5
Use valid type hints for Python 3.9
tristan957 Oct 10, 2024
b2ecbf3
Introduce "quota" ErrorKind (#9300)
awarus Oct 11, 2024
1849356
tests: stabilize test_storage_controller_heartbeats (#9347)
jcsp Oct 11, 2024
6baf1aa
proxy: Demote some errors to warnings in logs (#9354)
cloneable Oct 11, 2024
326cd80
ci: gh-workflow-stats-action v0.1.4: remove debug output and proper p…
fedordikarev Oct 11, 2024
091a175
Test versions mismatch (#9167)
a-masterov Oct 11, 2024
5ef805e
CI(run-python-test-set): allow to skip missing compatibility snapshot…
bayandin Oct 11, 2024
ab5bbb4
proxy: refactor auth backends (#9271)
conradludgate Oct 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 0 additions & 5 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@
!Cargo.toml
!Makefile
!rust-toolchain.toml
!scripts/combine_control_files.py
!scripts/ninstall.sh
!vm-cgconfig.conf
!docker-compose/run-tests.sh

# Directories
Expand All @@ -17,15 +15,12 @@
!compute_tools/
!control_plane/
!libs/
!neon_local/
!pageserver/
!patches/
!pgxn/
!proxy/
!storage_scrubber/
!safekeeper/
!storage_broker/
!storage_controller/
!trace/
!vendor/postgres-*/
!workspace_hack/
3 changes: 3 additions & 0 deletions .github/actions/run-python-test-set/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,9 @@ runs:
name: compatibility-snapshot-${{ runner.arch }}-${{ inputs.build_type }}-pg${{ inputs.pg_version }}
# Directory is created by test_compatibility.py::test_create_snapshot, keep the path in sync with the test
path: /tmp/test_output/compatibility_snapshot_pg${{ inputs.pg_version }}/
# The lack of compatibility snapshot shouldn't fail the job
# (for example if we didn't run the test for non build-and-test workflow)
skip-if-does-not-exist: true

- name: Upload test results
if: ${{ !cancelled() }}
Expand Down
18 changes: 16 additions & 2 deletions .github/actions/upload/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ inputs:
path:
description: "A directory or file to upload"
required: true
skip-if-does-not-exist:
description: "Allow to skip if path doesn't exist, fail otherwise"
default: false
required: false
prefix:
description: "S3 prefix. Default is '${GITHUB_SHA}/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}'"
required: false
Expand All @@ -15,10 +19,12 @@ runs:
using: "composite"
steps:
- name: Prepare artifact
id: prepare-artifact
shell: bash -euxo pipefail {0}
env:
SOURCE: ${{ inputs.path }}
ARCHIVE: /tmp/uploads/${{ inputs.name }}.tar.zst
SKIP_IF_DOES_NOT_EXIST: ${{ inputs.skip-if-does-not-exist }}
run: |
mkdir -p $(dirname $ARCHIVE)
Expand All @@ -33,14 +39,22 @@ runs:
elif [ -f ${SOURCE} ]; then
time tar -cf ${ARCHIVE} --zstd ${SOURCE}
elif ! ls ${SOURCE} > /dev/null 2>&1; then
echo >&2 "${SOURCE} does not exist"
exit 2
if [ "${SKIP_IF_DOES_NOT_EXIST}" = "true" ]; then
echo 'SKIPPED=true' >> $GITHUB_OUTPUT
exit 0
else
echo >&2 "${SOURCE} does not exist"
exit 2
fi
else
echo >&2 "${SOURCE} is neither a directory nor a file, do not know how to handle it"
exit 3
fi
echo 'SKIPPED=false' >> $GITHUB_OUTPUT
- name: Upload artifact
if: ${{ steps.prepare-artifact.outputs.SKIPPED == 'false' }}
shell: bash -euxo pipefail {0}
env:
SOURCE: ${{ inputs.path }}
Expand Down
19 changes: 9 additions & 10 deletions .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -193,16 +193,15 @@ jobs:
with:
submodules: true

# Disabled for now
# - name: Restore cargo deps cache
# id: cache_cargo
# uses: actions/cache@v4
# with:
# path: |
# !~/.cargo/registry/src
# ~/.cargo/git/
# target/
# key: v1-${{ runner.os }}-${{ runner.arch }}-cargo-clippy-${{ hashFiles('rust-toolchain.toml') }}-${{ hashFiles('Cargo.lock') }}
- name: Cache cargo deps
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
!~/.cargo/registry/src
~/.cargo/git
target
key: v1-${{ runner.os }}-${{ runner.arch }}-cargo-${{ hashFiles('./Cargo.lock') }}-${{ hashFiles('./rust-toolchain.toml') }}-rust

# Some of our rust modules use FFI and need those to be checked
- name: Get postgres headers
Expand Down
41 changes: 41 additions & 0 deletions .github/workflows/report-workflow-stats.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Report Workflow Stats

on:
workflow_run:
workflows:
- Add `external` label to issues and PRs created by external users
- Benchmarking
- Build and Test
- Build and Test Locally
- Build build-tools image
- Check Permissions
- Check build-tools image
- Check neon with extra platform builds
- Cloud Regression Test
- Create Release Branch
- Handle `approved-for-ci-run` label
- Lint GitHub Workflows
- Notify Slack channel about upcoming release
- Periodic pagebench performance test on dedicated EC2 machine in eu-central-1 region
- Pin build-tools image
- Prepare benchmarking databases by restoring dumps
- Push images to ACR
- Test Postgres client libraries
- Trigger E2E Tests
- cleanup caches by a branch
types: [completed]

jobs:
gh-workflow-stats:
name: Github Workflow Stats
runs-on: ubuntu-22.04
permissions:
actions: read
steps:
- name: Export GH Workflow Stats
uses: neondatabase/[email protected]
with:
DB_URI: ${{ secrets.GH_REPORT_STATS_DB_RW_CONNSTR }}
DB_TABLE: "gh_workflow_stats_neon"
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_RUN_ID: ${{ github.event.workflow_run.id }}
1 change: 1 addition & 0 deletions CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/compute_tools/ @neondatabase/control-plane @neondatabase/compute
/storage_controller @neondatabase/storage
/storage_scrubber @neondatabase/storage
/libs/pageserver_api/ @neondatabase/storage
/libs/postgres_ffi/ @neondatabase/compute @neondatabase/storage
/libs/remote_storage/ @neondatabase/storage
Expand Down
Loading
Loading