feat: check for panic in logs on start

[Reecepbcups]

closes #1238

Summary

This is a huge user experience improvement for ICT & local-ic upon a bad genesis param set (or some other failed startup option).

Before, the network would hang and just state it failed to query
:
failed to start chains: failed to start chain localosmosis-1: All attempts fail:
#1: post failed: Post "http://0.0.0.0:26659\": dial tcp 0.0.0.0:26659: connect: connection refused
...
#30 ...

Showcase

This chain uses a genesis modification of {"key": "app_state.gov.params.voting_period", "value": "bad"},. Both validators show this error in an easy to read format + the docker logs for the stacktrace.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
interchaintest-docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 16, 2024 3:36pm

[joelsmith-2019]

Will this slow down tests that spin up numerous containers? I just want to double check we won't wait 1 second for each ephemeral container we spin up.

[Reecepbcups]

@joelsmith-2019

Will this slow down tests that spin up numerous containers?

Test are only slowed down by the 1 second per container, and containers are spun up in parallel. So if you have 1 or 100 containers, the test time only slows down by 1 second

just want to double check we won't wait 1 second for each ephemeral container we spin up

No, ephemeral containers use their own job.Run() method which is outside of the containerLifecycle.

func (tn *ChainNode) Exec(ctx context.Context, cmd []string, env []string) ([]byte, []byte, error) {
	job := dockerutil.NewImage(tn.logger(), tn.DockerClient, tn.NetworkID, tn.TestName, tn.Image.Repository, tn.Image.Version)
	opts := dockerutil.ContainerOptions{
		Env:   env,
		Binds: tn.Bind(),
	}
	res := job.Run(ctx, cmd, opts)
	return res.Stdout, res.Stderr, res.Err
}

containerLifecycle is only used for long running containers (nodes & sidecars).

I had previously tested this with

	if err := c.CheckForFailedStart(ctx, time.Second*10); err != nil {
		return err
	}

Only time it hangs is post "Starting container" once.

[Reecepbcups]

I also have a WIP speed up branch which will reduce test time by like 40%. Theoretical max is ~90 - 92% with state cache. Which would make this 1 second start wait the slowest part of ICT in the future

#1208