Throttling can cause the rotation to hang and not rollback

[gregbreen]

First time this has been seen, in tens of thousands of certificate rotations. The device was throttled, when trying to get the job, when starting up. In this case, it had just restarted, and was trying to use the new certificate and key.

state_getting_job.py is expecting a get/accepted but it gets a get/rejected because of the throttle. It returns to state Idle, keeping the new certificate and key active (which results in Greengrass losing connectivity because the cloud times out and restores the old certificate). Instead, if the backup exists, state_getting_job.py should rollback.

In this situation, restarting Greengrass is enough to recover.

2024-10-15T11:56:48.151Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Starting the state machine. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.152Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic $aws/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/jobs/notify-next. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.194Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic $aws/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/jobs/+/get/accepted. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.231Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic $aws/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/jobs/+/get/rejected. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.262Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic $aws/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/jobs/+/update/accepted. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.304Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic $aws/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/jobs/+/update/rejected. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.337Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic awslabs/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/certificate/create/accepted. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.377Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic awslabs/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/certificate/create/rejected. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.412Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic awslabs/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/certificate/commit/accepted. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.450Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Subscribing to topic awslabs/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/certificate/commit/rejected. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.487Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Changing state to StateGettingJob. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.488Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Publishing message on topic $aws/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/jobs/$next/get: {}. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.506Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Received new message on topic $aws/things/GreengrassEC2DeviceFarm-ubuntu-22-04-x86-64/jobs/$next/get/rejected: {"timestamp":1728993408,"code":"RequestThrottled","message":"Rate exceeded"}. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}
2024-10-15T11:56:48.506Z [INFO] (Copier) aws.greengrass.labs.CertificateRotator: stdout. Changing state to StateIdle. {scriptName=services.aws.greengrass.labs.CertificateRotator.lifecycle.Run.Script, serviceName=aws.greengrass.labs.CertificateRotator, currentState=RUNNING}