feat: add support to record performance report of database

[tanmoysrt]

Related #723

Required agent PR frappe/agent#129

Changes -

• Added fetch_performance_report method in Database Server doctype
• Added Performance Report doctype which will contain performance schema reports
• Added all the reports (available in mysql workbench) as child table of Performance Report
• Anyone can enable/disable specific performance report from `Database Server > Mariadb Settings"
• Added cronjob to fetch reports every 15 minutes

Preview

2024-02-04.10-29-39.mp4

[codecov]

Codecov Report

Attention: Patch coverage is 5.36398% with 247 lines in your changes missing coverage. Please review.

Project coverage is 36.54%. Comparing base (f11ed4a) to head (86aa9ca).
Report is 190 commits behind head on master.

Files with missing lines	Patch %	Lines
...s/press/doctype/database_server/database_server.py	4.76%	140 Missing ⚠️
...ype/db_performance_report/db_performance_report.py	27.77%	13 Missing ⚠️
press/press/doctype/agent_job/agent_job.py	20.00%	4 Missing ⚠️
press/agent.py	25.00%	3 Missing ⚠️
...ctype/global_waits_by_time/global_waits_by_time.py	0.00%	3 Missing ⚠️
...r_stats_by_schema/innodb_buffer_stats_by_schema.py	0.00%	3 Missing ⚠️
...fer_stats_by_table/innodb_buffer_stats_by_table.py	0.00%	3 Missing ⚠️
...schema_index_statistics/schema_index_statistics.py	0.00%	3 Missing ⚠️
...schema_table_statistics/schema_table_statistics.py	0.00%	3 Missing ⚠️
...ffer/schema_table_statistics_with_innodb_buffer.py	0.00%	3 Missing ⚠️
... and 23 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1395      +/-   ##
==========================================
- Coverage   37.36%   36.54%   -0.82%     
==========================================
  Files         349      398      +49     
  Lines       29398    30861    +1463     
==========================================
+ Hits        10984    11278     +294     
- Misses      18414    19583    +1169     

Flag	Coverage Δ
	36.54% <5.36%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ankush]

@tanmoysrt awesome stuff!

I've some queries and suggestions.

1. How much time this takes to fetch from DB and roughly what's size of data per day (assuming 15 min interval)
2. Does this capture events_statemens_summary_by_digest or equivalent digest summaries? (found it to be most useful replacement for slow query logs)
3. If you implement summary by digest you'll need to keep truncating the stats and regenerate it on our end using historical data. The digest has some hard limit (10K queries) after which it stops capturing details and stores them against NULL digest.
4. Can we setup log clearing so this doctype and all child tables get cleared at certain interval. Ref: refactor!: improve/extend log settings frappe#17159 see how it's done for email queue. We will need fast clearing in raw SQL without touching ORM.

[tanmoysrt]

1. It takes 5~6 seconds to fetch from DB and then couple of seconds to fetch from server to press end. Although it may vary for large database as I was testing on just development setup. Although if we start checking Innodb buffer stats, it can slowdown perf schema query + normal queries according to docs
2. We are not collecting events_statements_summary_by_digest but we can add that. just a little change required. Check this PR, you will find out the sql statements of available reports, that we are fetching from DB
3. Need to check
4. Yup, we can setup log clearing, will check the PR.

cc @ankush

[tanmoysrt]

After getting some more context on the perf schema, understood that we don't need all the information actually.

Let's continue the discussion here - #2361 (comment)

[tanmoysrt]

Closing this PR.
Continuing the work in performance insights #2361.