[Core feature] Refactor distributed job using common ReplicaSpec

[troychiu]

Motivation: Why do you think this is important?

Right now, different types of distributed jobs such as tensorflow, PyTorch, ... all have their own ReplicaSpec. Based on this discussion thread #4179 (review), we can have a shared ReplicaSpec in common.proto so that all types of distributed jobs can leverage it.

Goal: What should the final outcome look like, ideally?

In common.proto, we have a ReplicaSpec like

# common.proto
message ReplicaSpec {
    // Number of replicas
    int32 replicas = 1;
    
    // Image used for the replica group
    string image = 2;
    
    // Resources required for the replica group
    core.Resources resources = 3;
    
    // Restart policy determines whether pods will be restarted when they exit
    RestartPolicy restart_policy = 4;
    ...
}

and all types of distributed jobs (tensorflow, PyTorch, ray, ...) share it.

Describe alternatives you've considered

Stay what we have now. That is, all types of distributed job have their own ReplicaSpec.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[hamersaw]

IMO there needs to be more discussion here. There are certainly many configuration parameters that will differ between ReplicaSpec for each of the distributed environments, and even within each environment. Is the RestartPolicy you have mocked applied to every plugin? If not, this is another parameter that may result in unintended behavior. Is the proposal to use this common.pb message as a field of each replica spec? For example:

// spark.pb
message SparkWorkerReplicaSpec {
    ReplicaSpec replica_spec = 1;
    
    ... spark sepcific configuration
}

// dask.pb
message DaskScheduler {
   RepliaSpec replica_spec = 1;
   
   ...
}

message DaskWorker {
    ReplicaSpec replica_spec = 1;
    
    ....
}

[MortalHappiness]

#take