rocksdb-8.10: pending compaction jobs and too many L0 files.

[LykxSassinator]

Bug Report

There exists numerous L0 files in WriteCF, and cannot be compacted into higher level ,which causes the throttling problems.

And according to the provided logs, we've found that there exists pending compaction jobs, hang on
some abnormal operations, without any Compacted logs to reflect these jobs are finished.

Meanwhile, there exists no other detailed logs about the above compaction jobs, while the flushing jobs are normal
making the number of L0 files in WriteCF growing without expectations.

Taking the following compaction jobs as examples:

260614
261003

~ ❯ grep "JOB 260614" rocksdb-2025-03-07T21-04-53.529.info                                                                                                                                                                                                                                                    ✘ INT at 11:49:49
[2025/03/07 10:22:39.456 +08:00][5][INFO] [db/compaction/compaction_job.cc:2082] [write] [JOB 260614] Compacting 5@0 + 46@3 files to L3, score 12.50

~ ❯ grep "JOB 261003" rocksdb-2025-03-07T21-04-53.529.info                                                                                                                                                                                                                                                          at 11:50:03
[2025/03/07 10:31:52.137 +08:00][5][INFO] [db/compaction/compaction_job.cc:2082] [write] [JOB 261003] Compacting 1@4 + 1@5 files to L5, score -1.00

But fortunately, the relevant node becomes normal after restarting.

What version of TiKV are you using?

v8.5.1

What operating system and CPU are you using?

Linux (Centos)

Steps to reproduce

Cannot reproduce.

What did you expect?

No pending compactions and the number of L0 files can decreased to the normal level (< 20).

What did happened?

There exists pending compactions jobs and the number of L0 files in WriteCF keeps growing, reaching 1000.