Add support of tpu-client-next to validator

[KirillLykov]

Problem

Current quic client implementation ConnectionCache has some disadvantages:

• creates fragmented traffic: if you send tx1, tx2 which are split into chunks c1_1, c1_2, c2_1, c2_2 than ConnectionCache doesn't guarantee that they will be in the order.
• uses internal tokio queue of futures instead of explicit channel. So when you call connection_cache.send what happens it puts future to the tokio internal queue. If there is a flow control on server side slowing down connection, this futures queue will grow on client side. There is not control over it.
• need to warm-up cache by sending empty packets in ConnectionCache. Instead, here we try opening connections to the next leaders before we use them.
• No way to cancel ConnectionCache gracefully.
• Uses unnecessary synchronization code internally, slowing down sending and creating not uniform flow in case of heavy load
• creates internally tokio runtime instead of taking one from outside.

Internally validator uses quic client for two components:

1. Forwarder
2. SendTransactionService

This issue to is track the work on adding support of new network crate tpu-client-next to the these components.
When finished, it should be possible to choose between two client codes.

Proposed Solution

• Extract network-related code from SendTransactionService into a separate structure Extract client code in send_transaction_service into a new structure #3423
• Add a new client support to SendTransactionService, instrument all the related tests with new client as well use tpu-client-next in send_transaction_service #3515
• Verify that backport PRs merged: v2.1: Extract client code in send_transaction_service into a new structure (backport of #3423) #3480 and (after) v2.1: use tpu-client-next in send_transaction_service (backport of #3515) #3884
• Add a new client to validator Use tpu client next in validator with receiver #3893
• Add tpu-client-next to ForwardingStage (new implementation of forwarding by Andrew), poc is poc for use tpu client next forwarding #3286
• Probably add some implementation of the same to the Forwarder since it is unclear when ForwardingStage will be released
• Add multi node fan out parameter to the tpu-client-next add fanout to tpu-client-next #3478
• Add equivalent of ConnectionCache warm-up service to the tpu-client-next add fanout to tpu-client-next #3478
• Add tests for set_identity Add set_identity tests #3794,
• Modify this test to work with flag use_tpu_client_next.

[KirillLykov]

Address #3515 (comment)

[KirillLykov]

Address #3515 (comment) (outside of backporting)

[KirillLykov]

Reapplying reverted PRs back to master

• Revert PR_3: Revert "send-transactions: optimize retry pool" #4250 . The original PR: send-transactions: optimize retry pool #31
• Revert PR_2: Revert "use tpu-client-next in send_transaction_service (#3515)" #4252 . The original PR: use tpu-client-next in send_transaction_service #3515
• Revert PR_1: Revert "Extract client code in send_transaction_service into a new structure" #4254 . The original PR: Extract client code in send_transaction_service into a new structure #3423
• Add back PR_1 (Part 1): Reapply changes from reverted PR 3423 #4257 with changes minimizing API changes, see aed704e
• Add back PR_1 (Part 2): Reapply changes from reverted PR 3423: move structures to new modules #4427
• Add backport for PR_1: v2.1: Reapply changes from reverted PR 3423 (backport of #4257) #4671
• Add back PR_2 (Part 1): modifications on tpu-client-next tpu-client-next: return receiver in scheduler::run #4454
• Add back PR_2 (Part 2): Reapply 3515: add get_not_unique_leader_tpus and depricate get_leader_tpus_with_slots #4753
• Add back PR_2 (Part 3): Reapply 3515: add tpu client next to sts #4758
• Add back PR_2 (Part 4): Reapply 3515: use SendTransactionService::new_with_client in rpc #5111
• Add backport for PR_3 Reapply PR#31: optimize retry pool #5113