Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rover dev high cpu after composition failed #1906

Open
benrea opened this issue May 22, 2024 · 3 comments
Open

rover dev high cpu after composition failed #1906

benrea opened this issue May 22, 2024 · 3 comments
Labels
bug 🐞 triage issues and PRs that need to be triaged

Comments

@benrea
Copy link

benrea commented May 22, 2024 •
edited
Loading

Description

When running rover dev cpu usage spikes to ~100% following composition failed i.e. all subgraphs removed, and remains high even if subgraphs & composition are restored.

💀 composition failed, killing the router
error: No subgraphs were found in the supergraph config.

Not specific to local development - also occurs with remote subgraph(s)
With multiple subgraphs, only occurs once all subgraphs removed

Doesn't occur if composition fails/no subgraphs were reachable on startup, though rover dev never retries in this scenario.
My desired setup is to run rover in docker for better local DX, which works but suffers the same issue.

Tested with rover 0.20.0, 0.21.0, 0.22.0, 0.23.0

Steps to reproduce

Example repo to reproduce with a single subgraph: https://github.com/benrea/rover-example
This mimics our setup using go + gqlgen + rover dev for local development against a federated graph.

Expected result

Cpu usage expected to spike during introspection(?) or genuine usage e.g. handling requests.

Actual result

Cpu usage spikes to ~100% and remains high, even after subgraphs/composition restored and operational.

Restarting rover / new process returns to normal.

Environment

Rover Info:
Version: 0.23.0
Install Location: /Users/benrea/.rover/bin/rover
OS: Mac OS 14.4.0 [64-bit]
Shell: /bin/zsh
@benrea benrea added bug 🐞 triage issues and PRs that need to be triaged labels May 22, 2024
@benrea
Copy link
Author

benrea commented May 22, 2024 •
edited
Loading

Hack workaround... add a dummy subgraph to supergraph.yaml also fixes the 'nothing on startup' issue 😅

Was curious if schema-configured subgraph (instead of introspected) had the same issue

# supergraph.yaml
federation_version: =2.7.0
subgraphs:
  api:
    schema:
      subgraph_url: http://localhost:8080/graphql
  dummy:
    routing_url: http://localhost:9999/dummy
    schema:
      file: ./dummy.graphqls
# dummy.graphqls
type Query {
    dummy: String
}

@nora-heydecker-axa
Copy link

I have the same issue (on MacOS with rover 0.26.2). Composition failed and it's using loads of CPU.
I ran sample over it and got the following result (I cut out all threads that are sleeping):

    5818 Thread_1953478: tokio-runtime-worker
    + 5818 thread_start  (in libsystem_pthread.dylib) + 8  [0x18ac9e0fc]
    +   5818 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x18aca32e4]
    +     5818 std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71  (in rover-0.26.2) + 52  [0x10356b260]
    +       5818 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h6804af946b5153b7  (in rover-0.26.2) + 164  [0x1034d029c]
    +         5818 std::sys::backtrace::__rust_begin_short_backtrace::h1fdd89e8b59c7bad  (in rover-0.26.2) + 192  [0x1034cfad8]
    +           5818 tokio::runtime::blocking::pool::Inner::run::h091621eb564b80dc  (in rover-0.26.2) + 216  [0x1034cd194]
    +             5818 tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h04ecf35e89e3f246  (in rover-0.26.2) + 84  [0x10300238c]
    +               5818 tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h7f202bb45ee39d68  (in rover-0.26.2) + 56  [0x102f9bc5c]
    +                 5346 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 240  [0x102fae534]
    +                 ! 4199 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 72  [0x10305ac68]
    +                 ! : 1638 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 116  [0x102f81c68]
    +                 ! : | 1403 pthread_mutex_lock  (in libsystem_pthread.dylib) + 112,12,...  [0x18ac9d944,0x18ac9d8e0,...]
    +                 ! : | 235 DYLD-STUB$$pthread_mutex_lock  (in rover-0.26.2) + 4  [0x1035b65e4]
    +                 ! : 1565 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 80,128,...  [0x102f81c44,0x102f81c74,...]
    +                 ! : 996 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 620  [0x102f81e60]
    +                 ! :   849 pthread_mutex_unlock  (in libsystem_pthread.dylib) + 120,32,...  [0x18ac9de90,0x18ac9de38,...]
    +                 ! :   147 DYLD-STUB$$pthread_mutex_unlock  (in rover-0.26.2) + 4  [0x1035b65fc]
    +                 ! 790 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 204,40,...  [0x10305acec,0x10305ac48,...]
    +                 ! 357 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 640,632,...  [0x102f81e74,0x102f81e6c,...]
    +                 241 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 240,244  [0x102fae534,0x102fae538]
    +                 231 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 208  [0x10305acf0]
    5818 Thread_2115645: tokio-runtime-worker
    + 5818 thread_start  (in libsystem_pthread.dylib) + 8  [0x18ac9e0fc]
    +   5818 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x18aca32e4]
    +     5818 std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71  (in rover-0.26.2) + 52  [0x10356b260]
    +       5818 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h6804af946b5153b7  (in rover-0.26.2) + 164  [0x1034d029c]
    +         5818 std::sys::backtrace::__rust_begin_short_backtrace::h1fdd89e8b59c7bad  (in rover-0.26.2) + 192  [0x1034cfad8]
    +           5818 tokio::runtime::blocking::pool::Inner::run::h091621eb564b80dc  (in rover-0.26.2) + 216  [0x1034cd194]
    +             5818 tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h04ecf35e89e3f246  (in rover-0.26.2) + 84  [0x10300238c]
    +               5818 tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h7f202bb45ee39d68  (in rover-0.26.2) + 56  [0x102f9bc5c]
    +                 5346 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 240  [0x102fae534]
    +                 ! 4213 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 72  [0x10305ac68]
    +                 ! : 1715 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 116  [0x102f81c68]
    +                 ! : | 1482 pthread_mutex_lock  (in libsystem_pthread.dylib) + 112,60,...  [0x18ac9d944,0x18ac9d910,...]
    +                 ! : | 233 DYLD-STUB$$pthread_mutex_lock  (in rover-0.26.2) + 4  [0x1035b65e4]
    +                 ! : 1525 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 80,156,...  [0x102f81c44,0x102f81c90,...]
    +                 ! : 972 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 620  [0x102f81e60]
    +                 ! : | 834 pthread_mutex_unlock  (in libsystem_pthread.dylib) + 32,120,...  [0x18ac9de38,0x18ac9de90,...]
    +                 ! : | 138 DYLD-STUB$$pthread_mutex_unlock  (in rover-0.26.2) + 4  [0x1035b65fc]
    +                 ! : 1 pthread_mutex_lock  (in libsystem_pthread.dylib) + 136  [0x18ac9d95c]
    +                 ! 762 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 204,40,...  [0x10305acec,0x10305ac48,...]
    +                 ! 371 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 640,632,...  [0x102f81e74,0x102f81e6c,...]
    +                 242 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 244,240  [0x102fae538,0x102fae534]
    +                 230 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 208  [0x10305acf0]
    5818 Thread_2123946: tokio-runtime-worker
    + 5818 thread_start  (in libsystem_pthread.dylib) + 8  [0x18ac9e0fc]
    +   5818 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x18aca32e4]
    +     5818 std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71  (in rover-0.26.2) + 52  [0x10356b260]
    +       5818 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h6804af946b5153b7  (in rover-0.26.2) + 164  [0x1034d029c]
    +         5818 std::sys::backtrace::__rust_begin_short_backtrace::h1fdd89e8b59c7bad  (in rover-0.26.2) + 192  [0x1034cfad8]
    +           5818 tokio::runtime::blocking::pool::Inner::run::h091621eb564b80dc  (in rover-0.26.2) + 216  [0x1034cd194]
    +             5818 tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h04ecf35e89e3f246  (in rover-0.26.2) + 84  [0x10300238c]
    +               5818 tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h7f202bb45ee39d68  (in rover-0.26.2) + 56  [0x102f9bc5c]
    +                 5391 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 240  [0x102fae534]
    +                 ! 4270 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 72  [0x10305ac68]
    +                 ! : 1749 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 116  [0x102f81c68]
    +                 ! : | 1516 pthread_mutex_lock  (in libsystem_pthread.dylib) + 112,32,...  [0x18ac9d944,0x18ac9d8f4,...]
    +                 ! : | 233 DYLD-STUB$$pthread_mutex_lock  (in rover-0.26.2) + 4  [0x1035b65e4]
    +                 ! : 1501 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 80,128,...  [0x102f81c44,0x102f81c74,...]
    +                 ! : 1020 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 620  [0x102f81e60]
    +                 ! :   865 pthread_mutex_unlock  (in libsystem_pthread.dylib) + 32,120,...  [0x18ac9de38,0x18ac9de90,...]
    +                 ! :   155 DYLD-STUB$$pthread_mutex_unlock  (in rover-0.26.2) + 4  [0x1035b65fc]
    +                 ! 790 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 204,20,...  [0x10305acec,0x10305ac34,...]
    +                 ! 331 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 640,632,...  [0x102f81e74,0x102f81e6c,...]
    +                 215 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 208  [0x10305acf0]
    +                 212 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 244,240  [0x102fae538,0x102fae534]
    5818 Thread_2127838: tokio-runtime-worker
    + 5818 thread_start  (in libsystem_pthread.dylib) + 8  [0x18ac9e0fc]
    +   5818 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x18aca32e4]
    +     5818 std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71  (in rover-0.26.2) + 52  [0x10356b260]
    +       5818 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h6804af946b5153b7  (in rover-0.26.2) + 164  [0x1034d029c]
    +         5818 std::sys::backtrace::__rust_begin_short_backtrace::h1fdd89e8b59c7bad  (in rover-0.26.2) + 192  [0x1034cfad8]
    +           5818 tokio::runtime::blocking::pool::Inner::run::h091621eb564b80dc  (in rover-0.26.2) + 216  [0x1034cd194]
    +             5818 tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h04ecf35e89e3f246  (in rover-0.26.2) + 84  [0x10300238c]
    +               5818 tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h7f202bb45ee39d68  (in rover-0.26.2) + 56  [0x102f9bc5c]
    +                 5333 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 240  [0x102fae534]
    +                 ! 4228 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 72  [0x10305ac68]
    +                 ! : 1630 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 116  [0x102f81c68]
    +                 ! : | 1398 pthread_mutex_lock  (in libsystem_pthread.dylib) + 112,32,...  [0x18ac9d944,0x18ac9d8f4,...]
    +                 ! : | 232 DYLD-STUB$$pthread_mutex_lock  (in rover-0.26.2) + 4  [0x1035b65e4]
    +                 ! : 1571 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 584,156,...  [0x102f81e3c,0x102f81c90,...]
    +                 ! : 1027 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 620  [0x102f81e60]
    +                 ! :   865 pthread_mutex_unlock  (in libsystem_pthread.dylib) + 32,120,...  [0x18ac9de38,0x18ac9de90,...]
    +                 ! :   162 DYLD-STUB$$pthread_mutex_unlock  (in rover-0.26.2) + 4  [0x1035b65fc]
    +                 ! 764 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 204,20,...  [0x10305acec,0x10305ac34,...]
    +                 ! 341 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 640,632,...  [0x102f81e74,0x102f81e6c,...]
    +                 254 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 208  [0x10305acf0]
    +                 231 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 244,240  [0x102fae538,0x102fae534]
    5818 Thread_2136765: tokio-runtime-worker
      5818 thread_start  (in libsystem_pthread.dylib) + 8  [0x18ac9e0fc]
        5818 _pthread_start  (in libsystem_pthread.dylib) + 136  [0x18aca32e4]
          5818 std::sys::pal::unix::thread::Thread::new::thread_start::h1bd1b9c95010bf71  (in rover-0.26.2) + 52  [0x10356b260]
            5818 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h6804af946b5153b7  (in rover-0.26.2) + 164  [0x1034d029c]
              5818 std::sys::backtrace::__rust_begin_short_backtrace::h1fdd89e8b59c7bad  (in rover-0.26.2) + 192  [0x1034cfad8]
                5818 tokio::runtime::blocking::pool::Inner::run::h091621eb564b80dc  (in rover-0.26.2) + 216  [0x1034cd194]
                  5818 tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h04ecf35e89e3f246  (in rover-0.26.2) + 84  [0x10300238c]
                    5818 tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h7f202bb45ee39d68  (in rover-0.26.2) + 56  [0x102f9bc5c]
                      5379 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 240  [0x102fae534]
                      ! 4338 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 72  [0x10305ac68]
                      ! : 1742 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 116  [0x102f81c68]
                      ! : | 1529 pthread_mutex_lock  (in libsystem_pthread.dylib) + 112,132,...  [0x18ac9d944,0x18ac9d958,...]
                      ! : | 213 DYLD-STUB$$pthread_mutex_lock  (in rover-0.26.2) + 4  [0x1035b65e4]
                      ! : 1530 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 80,40,...  [0x102f81c44,0x102f81c1c,...]
                      ! : 1066 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 620  [0x102f81e60]
                      ! :   927 pthread_mutex_unlock  (in libsystem_pthread.dylib) + 120,32,...  [0x18ac9de90,0x18ac9de38,...]
                      ! :   139 DYLD-STUB$$pthread_mutex_unlock  (in rover-0.26.2) + 4  [0x1035b65fc]
                      ! 745 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 40,204,...  [0x10305ac48,0x10305acec,...]
                      ! 296 crossbeam_channel::flavors::zero::Channel$LT$T$GT$::recv::haca1c4ef6d92e892  (in rover-0.26.2) + 640,632,...  [0x102f81e74,0x102f81e6c,...]
                      234 _$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he2990c49ed385a47  (in rover-0.26.2) + 244,240  [0x102fae538,0x102fae534]
                      205 crossbeam_channel::channel::Receiver$LT$T$GT$::recv::hd8073d4ec531a4b7  (in rover-0.26.2) + 208  [0x10305acf0]

It looks like these threads are spinning on crossbeam channels

@aaronArinder
Copy link
Contributor

thanks for the report @benrea and the investigation, @nora-heydecker-axa!

we're working on some related stuff right now and are keeping tabs on this issue to see if what we're doing resolves it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug 🐞 triage issues and PRs that need to be triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aaronArinder @benrea @nora-heydecker-axa