Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compute_ctl: Streamline and Pipeline startup SQL #9717

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Commits on Nov 11, 2024

  1. compute_ctl: Streamline and Pipeline startup SQL

    Before, compute_ctl didn't have a good registry for what command would run
    when, depending exclusively on sync code to apply changes. When users have
    many databases/roles to manage, this step can take a substantial amount of
    time, breaking assumptions about low (re)start times in other systems.
    
    This commit reduces the time compute_ctl takes to restart when changes must
    be applied, by making all commands more or less blind writes, and applying
    these commands in an asynchronous context, only waiting for completion once
    we know the commands have all been sent.
    
    Additionally, this reduces time spent by batching per-database operations
    where previously we would create a new SQL connection for every user-database
    operation we planned to execute.
    MMeent committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    bc9e0d6 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. Skip connection setup costs when the connection isn't going to be used.

    The previous iteration always connected to every database, which slowed down
    startup when most databases had no operations scheduled.
    MMeent committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    793d63b View commit details
    Browse the repository at this point in the history
  2. Move multiline/complex SQL statements into their own files

    Addresses @ tristan957's comment, and allows separate review of these queries.
    MMeent committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    88fe1bd View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Configuration menu
    Copy the full SHA
    6c582cf View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0f28a07 View commit details
    Browse the repository at this point in the history
  3. fixup! import lint

    MMeent committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    43af985 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. Reduce lock contention in parallel steps with RwLock

    Instead of Mutex, using an RwLock on shared modifyable state helps speed
    up read-only parallel steps (i.e. most parallel steps). This makes the
    per-database jobs actually work in parallel, rather than serialize on
    accessing the database- and roles tables.
    
    Additionally, this patch reduces the number of parallel workers to 1/3rd of
    max_connections, from max_connections - 10. With the previous setting I
    consistently hit issues with "ERROR: no unpinned buffers available", now
    it works like a charm.
    MMeent committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    8a35138 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c20e452 View commit details
    Browse the repository at this point in the history
  3. FMT, clippy

    MMeent committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    2320544 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d796cc2 View commit details
    Browse the repository at this point in the history