Dig into what caused prod slowness yesterday

[mahalakshme]

Context:

- Around 4pm-5pm on 13th March I was unable to execute SQL queries on prod db
- Some implementors reported timeout issues during sync
-  prod db EBSByteBalance got depleted on 13th March around the same time
- Tech4dev reported external API response time is extremely slow around the same time.
- From 'DB Load' RDS monitoring metric, looks like in the past 3 months, this could have happened around 8 times - around 1pm to 5pm - not sure if I have interpreted the numbers right
- Current activity in RDS was showing in red - something like the below:

Probable root causes:

- From DB looked like ETL was running parallely but from logs it was found sequential: https://github.com/avniproject/avni-etl/issues/91
- Tech4dev was trying to pull RWB data via Airbyte adapter from Avni to Dalgo
- This has been frequently the case and we are not aware

Probable solutions:

• Make ETL to read from prod db instead of read DB - I heard from some that this can save 50% of load on prod and some say this will not impact much

AC:

• How good is our prod DB performance during peak times (1pm - 6pm - correct this if I am wrong)? Do we need to action on it on priority?
• If we need to action, how frequently are we facing such prod db load issues
• What could be the probable solutions - would moving ETL to make it read from separate readDB help? or anything else?
• What are the probable root causes for such Avni prod DB overload issues?