BACK-END ANALYSIS: Overloaded & Inefficient Job Scheduling (Cron Jobs & Background Processes)

[divine-comedian]

Overloaded & Inefficient Job Scheduling (Cron Jobs & Background Processes)

Problem: The back-end runs too many resource-intensive cron jobs, leading to CPU spikes and performance degradation.

We should investigate this area of concern and provide analysis documentation to diagnose the back-end issues

• What does the data say? Logs, monitoring tools, historical usage data etc..
• Root Cause Analysis. Why does this issue occur? Where does this issue occur? What files, repos or dependencies are associated with it?
• Solution Recommendations. Short & long term fixes. What are the trade-offs if any

leave related documentation inside notion is here -> https://www.notion.so/giveth/Back-End-Improvements-1913ab28d48c80ff8dd7d4d643d2960a?pvs=4#1913ab28d48c8086b310d347fb62ddbe

[divine-comedian]

JFYI @Giveth/devops

[CarlosQ96]

@divine-comedian

CHECK_AND_UPDATE_ENDAOMENT_PROJECTS_CRONJOB_EXPRESSION: 0 0 1 * *
CHECK_PROJECT_VERIFICATION_STATUS_CRONJOB_EXPRESSION: 0 0 * * 3
CHECK_USERS_SUPER_TOKEN_BALANCES_CRONJOB_TIME: 0 0 * * *
DONATION_SAVE_BACKUP_CRONJOB_EXPRESSION: 0 */5 * * * *
GENERATE_SITEMAP_CRONJOB_EXPRESSION: 0 0 * * 0
FILL_POWER_SNAPSHOT_BALANCE_CRONJOB_EXPRESSION: 20 */5 * * * *
FILL_BLOCK_NUMBERS_OF_SNAPSHOTS_CRONJOB_EXPRESSION: 3 0 */4 * * *
IMPORT_LOST_DONATIONS_CRONJOB_EXPRESSION: */5 * * * *
INSTANT_BOOSTING_UPDATE_CRONJOB_EXPRESSION: 0 */5 * * * *
MATCH_DRAFT_DONATION_CRONJOB_EXPRESSION: 0 * * * *
REVIEW_OLD_GIV_PRICES_CRONJOB_EXPRESSION: 15 * * * *
SYNC_GIVING_BLOCKS_CRONJOB_EXPRESSION: 0 0 0 * * 0
SYNC_POIGN_ART_CRONJOB_EXPRESSION: 0 0 * * *
UPDATE_POWER_ROUND_CRONJOB_EXPRESSION: 1 0 */4 * * *
VERIFY_DONATION_CRONJOB_EXPRESSION: */5 * * * *
VERIFY_RECURRING_CRONJOB_EXPRESSION: 0 * * * *
VERIFY_RECURRING_DONATION_CRONJOB_EXPRESSION: 0 0 * * *
DB_POWER_BOOSTING_SNAPSHOT_CRONJOB_EXPRESSION: 13 */12 * * * *
MAKE_UNREVIEWED_PROJECT_LISTED_CRONJOB_EXPRESSION: 0 0 * * * *
SYNC_IDRISS_TWITTER_DONATIONS_CRONJOB_EXPRESSION: 0 0 0 1 9
UPDATE_RECURRING_DONATIONS_STREAM_CRONJOB: 0 0 * * *

this are all the cronjobs, they run closely to each other. Using crontab times. We can move some maybe 5 minutes apart as a first step to reduce cpu usage

[CarlosQ96]

Caches saved on redis

REFRESH_ESTIMATED_MATCHING_CACHE_DURATION: 120000
QF_ROUND_AND_MAIN_CATEGORIES_CACHE_DURATION: 600000
QF_ROUND_ESTIMATED_MATCHING_CACHE_DURATION: 120000

[CarlosQ96]

As a first step we should move the cronjobs to not all run together as it puts a lot of strain.

[CarlosQ96]

In the root we run, afterwards on donations. await refreshProjectEstimatedMatchingView();
I suggest we remove it, it slows down the execution of the app on startup

[CarlosQ96]

Suggest also removing threads library. It increases memory consumption drastically. But can do a comparison

[kkatusic]

@divine-comedian, @CarlosQ96 I added comments and table inside notion file

[RamRamez]

I can't see the logs, seems Grafana is not well.

@divine-comedian @CarlosQ96 is the problem with production or staging?
if it's with production, we can change the cronjob timings and spread heavy usage, like what we did for staging, it may alleviate the issue.

[kkatusic]

I can't see the logs, seems Grafana is not well.

@divine-comedian @CarlosQ96 is the problem with production or staging? if it's with production, we can change the cronjob timings and spread heavy usage, like what we did for staging, it may alleviate the issue.

same on mine side, no data, you should tag @geleeroyale and @mhmdksh

[divine-comedian]

Thanks team! Don't forget this issue is focused on documenting and ANALYSING the area of concern (Overloaded & Inefficient Job Scheduling) - provide any documentation, finds and/or proposed solutions into the notion article I provided above!

We do not need to execute any changes to our codebase as part of this issue.

[divine-comedian]

when you open user profile page it runs 3 queries about your profile, donation stats, information about project - runs 3 queries

can we batch those queries and make 1 request to the back-end.

kechy needs to find package name