error starting new script runner

[dosaboy]

I am seeing the following in the athena-processor typically a few days after a restart:

{"level":"info","msg":"Downloading 'uploads/sosreport-XXX.tar.xz' to '/tmp/athena/athena-report-sosreports865620253'","time":"2023-09-18T07:18:54Z"}
{"level":"error","msg":"Failed to get new runner: Not Found - http-code: 404","time":"2023-09-18T07:18:54Z"}
{"component":"pubsub","duration":266873658,"err":"Not Found - http-code: 404","handler":"athena-processor-30389025ce78","id":"436","level":"error","metadata":{"x-audit-user":""},"msg":"Failed Processing PubSub Msg","stack":"goroutine 23 [running]:
runtime/debug.Stack(0xb28700, 0xc0006c9a68, 0xbdb487)
/opt/hostedtoolcache/go/1.16.15/x64/src/runtime/debug/stack.go:24 +0x9f
github.com/lileio/pubsub/v2/middleware/logrus.Middleware.SubscribeInterceptor.func1(0xcc2990, 0xc000026030, 0xc0004b15ac, 0x3, 0xc000321560, 0xc00069c120, 0x105, 0x120, 0xc0003a60d8, 0xc0000bc8d0, ...)
/home/runner/go/pkg/mod/github.com/lileio/pubsub/[email protected]/middleware/logrus/logrus.go:57 +0x537
github.com/lileio/pubsub/v2.chainSubscriberMiddleware.func1.1(0xcc2990, 0xc000026030, 0xc0004b15ac, 0x3, 0xc000321560, 0xc00069c120, 0x105, 0x120, 0xc0003a60d8, 0xc0000bc8d0, ...)
/home/runner/go/pkg/mod/github.com/lileio/pubsub/[email protected]/subscribe.go:167 +0x170
github.com/lileio/pubsub/v2/providers/nats.(*Nats).Subscribe.func1(0xc00015a230)
/home/runner/go/pkg/mod/github.com/lileio/pubsub/[email protected]/providers/nats/nats.go:120 +0x58b
github.com/nats-io/stan%2ego.(*conn).processMsg(0xc00054a000, 0xc000276120)
/home/runner/go/pkg/mod/github.com/nats-io/[email protected]/stan.go:879 +0x2f9
github.com/nats-io/nats%2ego.(*Conn).waitForMsgs(0xc00054c000, 0xc00041c0c0)
/home/runner/go/pkg/mod/github.com/nats-io/[email protected]/nats.go:2412 +0x342
created by github.com/nats-io/nats%2ego.(*Conn).subscribeLocked
/home/runner/go/pkg/mod/github.com/nats-io/[email protected]/nats.go:3491 +0x4a5
","time":"2023-09-18T07:18:54Z","topic":"sosreports"}

This seems to result in the failure to run any further scripts until a manual restart is performed.

[dosaboy]

error is happening at

athena-core/pkg/processor/processor.go

Line 330 in 368a4cc

log.Errorf("Failed to get new runner: %s", err)

[nicolasbock]

What do you mean with that no further scripts are run? In my local testing I can still add files to the uploads folder and athena picks those up.