CIS doesn't gracefully handle 503 error from F5

[avinashchundu9]

Setup Details

CIS Version : 2.19
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP v17.1.1.3
AS3 Version: 3.52
Agent Mode: AS3
Orchestration: K8S
Orchestration Version:
Pool Mode: Cluster/Nodeport
Additional Setup details: Rancher/Calico network

Description

When F5 is replying back with 503 error due to high CPU/memory usage or when CIS post large declarations. CIS still keeps posting configuration and makes the problem even worst

Steps To Reproduce

1. Post large declarations to F5 until it get busy.
2. Check logs on CIS you will notice 503 error but CIS still try to post every 30 sec.

Expected Result

CIS to handle these issues more gracefully. Having incremental posting will help during issues not only for error 503 but also other issues. Post 30sec->60sec->120sec->240sec... Posting incrementally during issues or when F5 is busy will help handle problem better and gives time for F5 to recover.

Actual Result

Posting every 30 sec's.

Diagnostic Information

<Configuration files, error messages, logs>
Note: Sanitize the data. For example, be mindful of IPs, ports, application names and URLs
Note: The following F5 article outlines the information required when opening an issue.
https://support.f5.com/csp/article/K60974137

Observations (if any)

[trinaths]

Created [CONTCNTR-5239] for internal tracking.