-
Notifications
You must be signed in to change notification settings - Fork 442
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compute_ctl: Streamline and Pipeline startup SQL #9717
base: main
Are you sure you want to change the base?
Commits on Nov 11, 2024
-
compute_ctl: Streamline and Pipeline startup SQL
Before, compute_ctl didn't have a good registry for what command would run when, depending exclusively on sync code to apply changes. When users have many databases/roles to manage, this step can take a substantial amount of time, breaking assumptions about low (re)start times in other systems. This commit reduces the time compute_ctl takes to restart when changes must be applied, by making all commands more or less blind writes, and applying these commands in an asynchronous context, only waiting for completion once we know the commands have all been sent. Additionally, this reduces time spent by batching per-database operations where previously we would create a new SQL connection for every user-database operation we planned to execute.
Configuration menu - View commit details
-
Copy full SHA for bc9e0d6 - Browse repository at this point
Copy the full SHA bc9e0d6View commit details
Commits on Nov 12, 2024
-
Skip connection setup costs when the connection isn't going to be used.
The previous iteration always connected to every database, which slowed down startup when most databases had no operations scheduled.
Configuration menu - View commit details
-
Copy full SHA for 793d63b - Browse repository at this point
Copy the full SHA 793d63bView commit details -
Move multiline/complex SQL statements into their own files
Addresses @ tristan957's comment, and allows separate review of these queries.
Configuration menu - View commit details
-
Copy full SHA for 88fe1bd - Browse repository at this point
Copy the full SHA 88fe1bdView commit details
Commits on Nov 13, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 6c582cf - Browse repository at this point
Copy the full SHA 6c582cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f28a07 - Browse repository at this point
Copy the full SHA 0f28a07View commit details -
Configuration menu - View commit details
-
Copy full SHA for 43af985 - Browse repository at this point
Copy the full SHA 43af985View commit details
Commits on Nov 15, 2024
-
Reduce lock contention in parallel steps with RwLock
Instead of Mutex, using an RwLock on shared modifyable state helps speed up read-only parallel steps (i.e. most parallel steps). This makes the per-database jobs actually work in parallel, rather than serialize on accessing the database- and roles tables. Additionally, this patch reduces the number of parallel workers to 1/3rd of max_connections, from max_connections - 10. With the previous setting I consistently hit issues with "ERROR: no unpinned buffers available", now it works like a charm.
Configuration menu - View commit details
-
Copy full SHA for 8a35138 - Browse repository at this point
Copy the full SHA 8a35138View commit details -
Configuration menu - View commit details
-
Copy full SHA for c20e452 - Browse repository at this point
Copy the full SHA c20e452View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2320544 - Browse repository at this point
Copy the full SHA 2320544View commit details -
Configuration menu - View commit details
-
Copy full SHA for d796cc2 - Browse repository at this point
Copy the full SHA d796cc2View commit details