refactor: move rate limiter to zetaclient and add metrics

[ws4charlie]

Description

• move rate limiter logic to zetaclient. The method is named as ApplyRateLimiter
• leave query part in zetacore to be performant. The method is named as RateLimiterInput.
• order pending cctxs by height across all chains to be first-come-first-serve
• add metrics of percentage_of_rate_reached to monitor current rate.

Closes: 2074
Closes: 2082
Closes: 2092
Closes: 2093

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Include instructions and any relevant details so others can reproduce.

• Tested CCTX in localnet
• Tested in development environment
• Go unit tests
• Go integration tests
• Tested via GitHub Actions

Checklist:

• I have added unit tests that prove my fix feature works

[github-actions]

!!!WARNING!!!
nosec detected in the following files: x/crosschain/keeper/grpc_query_cctx_rate_limit.go, x/crosschain/keeper/grpc_query_cctx_rate_limit_test.go, zetaclient/zetacore_observer.go, zetaclient/zetacore_observer_test.go

Be very careful about using #nosec in code. It can be a quick way to suppress security warnings and move forward with development, it should be employed with caution. Suppressing warnings with #nosec can hide potentially serious vulnerabilities. Only use #nosec when you're absolutely certain that the security issue is either a false positive or has been mitigated in another way.

Only suppress a single rule (or a specific set of rules) within a section of code, while continuing to scan for other problems. To do this, you can list the rule(s) to be suppressed within the #nosec annotation, e.g: /* #nosec G401 */ or //#nosec G201 G202 G203
Broad #nosec annotations should be avoided, as they can hide other vulnerabilities. The CI will block you from merging this PR until you remove #nosec annotations that do not target specific rules.

Pay extra attention to the way #nosec is being used in the files listed above.

[codecov]

Codecov Report

Attention: Patch coverage is 84.19540% with 55 lines in your changes are missing coverage. Please review.

Project coverage is 72.30%. Comparing base (b460556) to head (ebc87ee).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2110      +/-   ##
===========================================
+ Coverage    71.87%   72.30%   +0.43%     
===========================================
  Files          245      247       +2     
  Lines        13730    13921     +191     
===========================================
+ Hits          9868    10066     +198     
+ Misses        3405     3394      -11     
- Partials       457      461       +4     

Files	Coverage Δ
pkg/math/float.go	100.00% <100.00%> (ø)
x/crosschain/keeper/cctx_utils.go	100.00% <100.00%> (ø)
x/crosschain/keeper/rate_limiter_flags.go	100.00% <100.00%> (ø)
zetaclient/core_context/zeta_core_context.go	91.45% <100.00%> (+0.62%)	⬆️
zetaclient/metrics/metrics.go	75.00% <ø> (ø)
zetaclient/ratelimiter/rate_limiter.go	100.00% <100.00%> (ø)
x/crosschain/keeper/grpc_query_cctx_rate_limit.go	94.92% <95.55%> (-1.16%)	⬇️
x/crosschain/types/rate_limiter_flags.go	93.54% <90.90%> (-6.46%)	⬇️
zetaclient/zetabridge/query.go	67.27% <87.27%> (+0.93%)	⬆️
zetaclient/zetacore_observer.go	27.31% <42.85%> (+10.94%)	⬆️

[skosito]

looks good, just some comments to consider before merging, mostly refactoring suggestions