[CELEBORN-1645] Introduce ShuffleFallbackPolicy to support custom implementation of shuffle fallback policy for CelebornShuffleFallbackPolicyRunner

[SteNicholas]

What changes were proposed in this pull request?

Introduce ShuffleFallbackPolicy to support custom implementation of shuffle fallback policy for CelebornShuffleFallbackPolicyRunner.

Why are the changes needed?

Shuffle fallback policy is fixed in CelebornShuffleFallbackPolicyRunner, which could not support user-defined fallback policy implementation. It's proposed to introduce ShuffleFallbackPolicy to support custom implementation of shuffle fallback policy for CelebornShuffleFallbackPolicyRunner.

Does this PR introduce any user-facing change?

Introduce ShuffleFallbackPolicy interface to determine whether fallback to vanilla Spark built-in shuffle implementation.

/**
 * The shuffle fallback policy determines whether fallback to vanilla Spark built-in shuffle
 * implementation.
 */
public interface ShuffleFallbackPolicy {

  /**
   * Returns whether fallback to vanilla spark built-in shuffle implementation.
   *
   * @param shuffleDependency The shuffle dependency of Spark.
   * @param celebornConf The configuration of Celeborn.
   * @param lifecycleManager The {@link LifecycleManager} of Celeborn.
   * @return Whether fallback to vanilla spark built-in shuffle implementation.
   */
  boolean needFallback(
      ShuffleDependency<?, ?, ?> shuffleDependency,
      CelebornConf celebornConf,
      LifecycleManager lifecycleManager);
}

How was this patch tested?

SparkShuffleManagerSuite

[SteNicholas]

Ping @FMX, @RexXiong.

[photogamerun]

it is Awsome @SteNicholas

[RexXiong]

LGTM, merge to main(v0.6.0)