Use new batch status api /status-job

[shtripat]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

Motivation and Context

How to test this PR?

run a batch replication and then mc batch status ALIAS JOB-ID and mc batch status ALIAS JOB-ID --json
For a in progress job it would output as

{
 "status": "in-progress",
 "metric": {
  "jobID": "replicate-QanJ6Lu4XNvdem6X4nBmUW:-1",
  "jobType": "replicate",
  "startTime": "2024-07-02T13:02:28.265715763Z",
  "lastUpdate": "0001-01-01T00:00:00Z",
  "retryAttempts": 10,
  "complete": false,
  "failed": false,
  "replicate": {
   "lastBucket": "testbversion",
   "lastObject": "hugefile",
   "objects": 1,
   "objectsFailed": 0,
   "bytesTransferred": 538888897,
   "bytesFailed": 0
  }
 }
}

and

For a completed job the outputs would looks like

{
 "status": "success",
 "metric": {
  "jobID": "replicate-UbKXvJTd768FEJG47phaHT:-1",
  "jobType": "replicate",
  "startTime": "2024-07-02T13:08:28.409982405Z",
  "lastUpdate": "2024-07-02T13:11:38.329332522Z",
  "retryAttempts": 1,
  "complete": true,
  "failed": false,
  "replicate": {
   "lastBucket": "testbversion",
   "lastObject": "uberfile",
   "objects": 2,
   "objectsFailed": 0,
   "bytesTransferred": 5907598017,
   "bytesFailed": 0
  }
 }
}

and

./mc batch status m1 "replicate-UbKXvJTd768FEJG47phaHT:-1"
✔ ✔ ✔ 
JobType:        replicate     
Objects:        2             
Versions:       2             
FailedObjects:  0             
Throughput:     30 MiB/s      
IOPs:           0.01 objs/s   
Transferred:    5.5 GiB       
Elapsed:        3m9.628075083s
CurrObjName:   	uberfile      	

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here

[shtripat]

Needs: minio/minio#19679

[vadmeste]

@shtripat I think we should keep client.Metrics() batch because it is much less expensive than client.BatchJobStatus, also client.Metrics() provides realtime information and it is better to keep it.

However client.BatchJobStatus() is useful when the job finishes, where client.Metrics() can be cleared (because it is in-memory server side data).

So overall, I think it is enough to add client.BatchJobStatus() only when the job is finished (nosuchJob in line 66:84)

[shtripat]

@shtripat I think we should keep client.Metrics() batch because it is much less expensive than client.BatchJobStatus, also client.Metrics() provides realtime information and it is better to keep it.

However client.BatchJobStatus() is useful when the job finishes, where client.Metrics() can be cleared (because it is in-memory server side data).

So overall, I think it is enough to add client.BatchJobStatus() only when the job is finished (nosuchJob in line 66:84)

@vadmeste do you mean a change as below

diff --git a/cmd/batch-status.go b/cmd/batch-status.go
index 67d2d7de..10b8dbac 100644
--- a/cmd/batch-status.go
+++ b/cmd/batch-status.go
@@ -2,6 +2,7 @@ package cmd
 
 import (
        "context"
+       "errors"
        "fmt"
        "strings"
        "time"
@@ -79,6 +80,8 @@ func mainBatchStatus(ctx *cli.Context) error {
        ctxt, cancel := context.WithCancel(globalContext)
        defer cancel()
 
+       ui := tea.NewProgram(initBatchJobMetricsUI(jobID))
+
        _, e := client.DescribeBatchJob(ctxt, jobID)
        nosuchJob := madmin.ToErrorResponse(e).Code == "XMinioAdminNoSuchJob"
        if nosuchJob {
@@ -86,26 +89,57 @@ func mainBatchStatus(ctx *cli.Context) error {
                if !globalJSON {
                        console.Infoln("Unable to find an active job, attempting to list from previously run jobs")
                }
-       }
-       fatalIf(probe.NewError(e), "Unable to lookup job status")
-
-       ui := tea.NewProgram(initBatchJobMetricsUI(jobID))
-       go func() {
-               res, e := client.BatchJobStatus(ctxt, jobID)
-               fatalIf(probe.NewError(e), "Unable to lookup job status")
-               if globalJSON {
-                       printMsg(batchJobStatusMessage{
-                               Status: "success",
-                               Metric: res.LastMetric,
+               go func() {
+                       res, e := client.BatchJobStatus(ctxt, jobID)
+                       fatalIf(probe.NewError(e), "Unable to lookup job status")
+                       if globalJSON {
+                               printMsg(batchJobStatusMessage{
+                                       Status: "success",
+                                       Metric: res.LastMetric,
+                               })
+                               if res.LastMetric.Complete || res.LastMetric.Failed {
+                                       cancel()
+                                       return
+                               }
+                       } else {
+                               ui.Send(res.LastMetric)
+                       }
+               }()
+       } else {
+               go func() {
+                       opts := madmin.MetricsOptions{
+                               Type:     madmin.MetricsBatchJobs,
+                               ByJobID:  jobID,
+                               Interval: time.Second,
+                       }
+                       e := client.Metrics(ctxt, opts, func(metrics madmin.RealtimeMetrics) {
+                               if globalJSON {
+                                       if metrics.Aggregated.BatchJobs == nil {
+                                               cancel()
+                                               return
+                                       }
+
+                                       job, ok := metrics.Aggregated.BatchJobs.Jobs[jobID]
+                                       if !ok {
+                                               cancel()
+                                               return
+                                       }
+
+                                       printMsg(metricsMessage{RealtimeMetrics: metrics})
+                                       if job.Complete || job.Failed {
+                                               cancel()
+                                               return
+                                       }
+                               } else {
+                                       ui.Send(metrics)
+                               }
                        })
-                       if res.LastMetric.Complete || res.LastMetric.Failed {
-                               cancel()
-                               return
+                       if e != nil && !errors.Is(e, context.Canceled) {
+                               fatalIf(probe.NewError(e).Trace(ctx.Args()...), "Unable to get current batch status")
                        }
-               } else {
-                       ui.Send(res.LastMetric)
-               }
-       }()
+               }()
+       }
+       fatalIf(probe.NewError(e), "Unable to lookup job status")
 
        if !globalJSON {
                if _, e := ui.Run(); e != nil {

[shtripat]

Its not working as expected and would need more work.

[vadmeste]

LGTM