Alertmanager goroutine leak test flakes

[seizethedave]

Describe the bug

I encountered a test flake in CI unrelated to my PR.
Here's the CI job.

--- FAIL: TestDistributor_DistributeRequest (21.47s)
    leak.go:14: found unexpected goroutines:
        [Goroutine 427 in state chan receive, with github.com/oklog/run.(*Group).Run on top of the stack:
        github.com/oklog/run.(*Group).Run(0xc00095cf10)
        	/__w/mimir/mimir/vendor/github.com/oklog/run/group.go:43 +0x251
        github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run(0xc0009de850)
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:108 +0x505
        created by github.com/grafana/mimir/pkg/alertmanager.(*Alertmanager).ApplyConfig in goroutine 217
        	/__w/mimir/mimir/pkg/alertmanager/alertmanager.go:544 +0x228b
         Goroutine 353 in state select, with github.com/prometheus/alertmanager/inhibit.(*Inhibitor).run on top of the stack:
        github.com/prometheus/alertmanager/inhibit.(*Inhibitor).run(0xc0009de850, {0x2cb9b58, 0xc001197900})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:[65](https://github.com/grafana/mimir/actions/runs/11733801665/job/32688750912#step:8:66) +0x1d9
        github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run.func1()
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:102 +0x47
        github.com/oklog/run.(*Group).Run.func1({0xc0004e5340?, 0xc00119f1e0?})
        	/__w/mimir/mimir/vendor/github.com/oklog/run/group.go:38 +0x3a
        created by github.com/oklog/run.(*Group).Run in goroutine 427
        	/__w/mimir/mimir/vendor/github.com/oklog/run/group.go:37 +0xa5
        ]
--- FAIL: TestMultitenantAlertmanager_loadAndSyncConfigs (0.94s)
    leak.go:14: found unexpected goroutines:
        [Goroutine 427 in state chan receive, with github.com/oklog/run.(*Group).Run on top of the stack:
        github.com/oklog/run.(*Group).Run(0xc00095cf10)
        	/__w/mimir/mimir/vendor/github.com/oklog/run/group.go:43 +0x251
        github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run(0xc0009de850)
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:108 +0x505
        created by github.com/grafana/mimir/pkg/alertmanager.(*Alertmanager).ApplyConfig in goroutine 217
        	/__w/mimir/mimir/pkg/alertmanager/alertmanager.go:544 +0x228b
         Goroutine 353 in state select, with github.com/prometheus/alertmanager/inhibit.(*Inhibitor).run on top of the stack:
        github.com/prometheus/alertmanager/inhibit.(*Inhibitor).run(0xc0009de850, {0x2cb9b58, 0xc00119[79](https://github.com/grafana/mimir/actions/runs/11733801665/job/32688750912#step:8:80)00})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:65 +0x1d9
        github.com/prometheus/alertmanager/inhibit.(*Inhibitor).Run.func1()
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/inhibit/inhibit.go:102 +0x47
        github.com/oklog/run.(*Group).Run.func1({0xc0004e5340?, 0xc00119f1e0?})
        	/__w/mimir/mimir/vendor/github.com/oklog/run/group.go:38 +0x3a
        created by github.com/oklog/run.(*Group).Run in goroutine 427
        	/__w/mimir/mimir/vendor/github.com/oklog/run/group.go:37 +0xa5
        ]

Environment

• GitHub actions.

[dimitarvdimitrov]

happened again

Details

--- FAIL: TestMultitenantAlertmanager_loadAndSyncConfigs (1.01s)
    leak.go:14: found unexpected goroutines:
        [Goroutine 861 in state semacquire, with sync.runtime_Semacquire on top of the stack:
        sync.runtime_Semacquire(0xc001124a58?)
        	/usr/local/go/src/runtime/sema.go:71 +0x25
        sync.(*WaitGroup).Wait(0xc001124a50)
        	/usr/local/go/src/sync/waitgroup.go:118 +0xa5
        github.com/prometheus/alertmanager/notify.FanoutStage.Exec({0xc00016c520, 0x1, 0xc000cfff40?}, {0x2cc6d20, 0xc000a5ab70}, {0x2cb2120, 0xc000bf1540}, {0xc0004e1f78, 0x1, 0x1})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/notify/notify.go:504 +0x390
        github.com/prometheus/alertmanager/notify.MultiStage.Exec({0xc000a1ca20, 0x6, 0x4ad125?}, {0x2cc6d20, 0xc000a5ab70}, {0x2cb2120, 0xc000bf1540}, {0xc0004e1f68, 0x1, 0x1})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/notify/notify.go:476 +0x169
        github.com/prometheus/alertmanager/notify.RoutingStage.Exec(0xc0009fb2c0, {0x2cc6d20, 0xc000a5ab70}, {0x2cb2120, 0xc000bf1540}, {0xc0004e1f68, 0x1, 0x1})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/notify/notify.go:462 +0x1b0
        github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert.func1({0x2cc6d20, 0xc000a5ab10}, {0xc0004e1f68, 0x1, 0x1})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:381 +0x64b
        github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run.func1({0xc0004e1f68, 0x1, 0x1})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:498 +0x6c
        github.com/prometheus/alertmanager/dispatch.(*aggrGroup).flush(0xc000fd0cf0, 0xc000eefeb8)
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:558 +0x6b5
        github.com/prometheus/alertmanager/dispatch.(*aggrGroup).run(0xc000fd0cf0, 0xc000a4d8f0)
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:497 +0x7a5
        created by github.com/prometheus/alertmanager/dispatch.(*Dispatcher).processAlert in goroutine 919
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:373 +0x7eb
         Goroutine 922 in state chan send, with github.com/grafana/mimir/pkg/alertmanager.(*state).broadcast on top of the stack:
        github.com/grafana/mimir/pkg/alertmanager.(*state).broadcast(...)
        	/__w/mimir/mimir/pkg/alertmanager/state_replication.go:312
        github.com/grafana/mimir/pkg/alertmanager.(*stateChannel).Broadcast(0xc00165afa8, {0xc00133a600, 0x60, 0x80})
        	/__w/mimir/mimir/pkg/alertmanager/state_replication.go:325 +0x1a5
        github.com/prometheus/alertmanager/nflog.(*Log).Log(0xc000e8ccc0, 0xc000bf[104](https://github.com/grafana/mimir/actions/runs/12129315606/job/33817507273?pr=10059#step:8:105)0, {0xc000a9f480, 0x1b}, {0xc001124a68, 0x1, 0x1}, {0x3b01340, 0x0, 0x0}, ...)
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/nflog/nflog.go:417 +0x76e
        github.com/prometheus/alertmanager/notify.SetNotifiesStage.Exec({{0x2cba100?, 0xc000e8ccc0?}, 0xc000bf1040?}, {0x2cc6d20, 0xc000a5abd0}, {0xc000f07da0?, 0x441fb4?}, {0xc0004e1f78, 0x1, 0x1})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/notify/notify.go:935 +0x253
        github.com/prometheus/alertmanager/notify.MultiStage.Exec({0xc0009bfc80, 0x4, 0xc0013d0f90?}, {0x2cc6d20, 0xc000a5ab70}, {0x2cb2120, 0xc000bf1540}, {0xc0004e1f78, 0x1, 0x1})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/notify/notify.go:476 +0x169
        github.com/prometheus/alertmanager/notify.FanoutStage.Exec.func1({0x2cb2240, 0xc000e55e90})
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/notify/notify.go:498 +0xc7
        created by github.com/prometheus/alertmanager/notify.FanoutStage.Exec in goroutine 861
        	/__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/notify/notify.go:497 +0xf0
        ]

cc @grafana/mimir-ruler-and-alertmanager-maintainers