Automatically restart runner after a crash

[kerem0comert]

Feature Description

Hello,

I have an Action Runner, which crashes every now and then due to some exception. For instance the latest logs show:

[Python Code Checks/code-check] ⭐ Run Post Checkout repository
[Python Code Checks/code-check] [DEBUG] run post step for 'Checkout repository'
[Python Code Checks/code-check] [DEBUG] executing remote job container: [node /var/run/act/actions/actions-checkout@v2/dist/index.js]
[Python Code Checks/code-check] Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/images/catthehacker/ubuntu:act-latest/json": context canceled
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xb5e0e1]
 
goroutine 434942 [running]:

where this container is the one that I use for Python-based Actions, the one you can find here.

Once any kind of crash such as this one happens, the Runner goes Offline and requires a manual restart. Is there a recommended way to auto-restart the runner? I guess one can setup a cronjob or something to check the runner is alive routinely, but I wanted to create this issue anyway to see if this is a problem that others face too or what's the best way to approach it.

Thanks!

Screenshots

No response

[lunny]

You can run it with systemd or docker then you can make it restart easier.

Can you paste more details so that we can know what made the runner panic.

[kerem0comert]

I see. Here are more detailed logs:

[Python Code Checks/code-check] [DEBUG] evaluating expression 'always()'

[Python Code Checks/code-check] [DEBUG] expression 'always()' evaluated to 'true'

[Python Code Checks/code-check] ⭐ Run Post Checkout repository

[Python Code Checks/code-check] [DEBUG] run post step for 'Checkout repository'

[Python Code Checks/code-check] [DEBUG] executing remote job container: [node /var/run/act/actions/actions-checkout@v2/dist/index.js]

[Python Code Checks/code-check] Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/images/catthehacker/ubuntu:act-latest/json": context canceled

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xb5e0e1]
 
goroutine 338871 [running]:

github.com/nektos/act/pkg/container.(*containerReference).UpdateFromImageEnv.(*containerReference).extractFromImageEnv.func1({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/container/docker_run.go:528 +0x1e1

github.com/nektos/act/pkg/container.(*containerReference).UpdateFromImageEnv.Executor.IfNot.func2({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:166 +0x4f

github.com/nektos/act/pkg/runner.(*RunContext).ApplyExtraPath(0xc0007de7e0, {0xefd3a8, 0xc000314150}, 0xc0000e27b8)

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/runner/run_context.go:416 +0x204

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.runPostStep.func1({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/runner/action.go:656 +0x891

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.runStepExecutor.func2({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/runner/step.go:141 +0xd83

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.Executor.If.func4({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:156 +0x45

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.Executor.If.func6({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:156 +0x45

github.com/nektos/act/pkg/runner.newJobExecutor.useStepLogger.func12({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/runner/job_executor.go:225 +0x2e2

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func14({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:182 +0x31

github.com/nektos/act/pkg/runner.newJobExecutor.func7({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/runner/job_executor.go:159 +0xe2

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func16({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func17({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:182 +0x31

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func18({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:182 +0x31

github.com/nektos/act/pkg/runner.newJobExecutor.NewPipelineExecutor.Executor.Then.func21({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:148 +0x142

github.com/nektos/act/pkg/runner.(*RunContext).Executor.func1({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/runner/run_context.go:595 +0x50

github.com/nektos/act/pkg/runner.(*runnerImpl).NewPlanExecutor.func1.1({0xefd3a8, 0xc000b5a240})

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/runner/runner.go:214 +0x19a

github.com/nektos/act/pkg/runner.(*runnerImpl).NewPlanExecutor.func1.NewParallelExecutor.func2.1(0x8f4ba5?, 0xc00c67bb00?)

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:107 +0x52

created by github.com/nektos/act/pkg/runner.(*runnerImpl).NewPlanExecutor.func1.NewParallelExecutor.func2 in goroutine 338870

        /go/pkg/mod/gitea.com/gitea/[email protected]/pkg/common/executor.go:105 +0xf7
[Python Code Checks/code-check] Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/images/catthehacker/ubuntu:act-latest/json": context canceled

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xb5e0e1]

[wolfogre]

@kerem0comert Which version of act runner are you using?

I think I have already fixed the panic: nektos/act#2071

[kerem0comert]

I run the latest via:

gitea/act_runner:latest

[wolfogre]

I run the latest via:

gitea/act_runner:latest

latest cannot identify the version; it could be very old if you pulled it a long time ago. You can go to /admin/actions/runners to check the version of runners.

[kerem0comert]

I see. Well in that case, the version seems to be v0.2.6.

[wolfogre]

I see. Well in that case, the version seems to be v0.2.6.

It's quite an old version; you can upgrade it. And as Lunny said, run it with systemd or Docker, then you can restart it more easily.