DMT: dmt command hangs when migration cluster fails to connect to source or target [CLC-432]

[sumnerib]

Given a running migration cluster, if the migration is started and the migration cluster is unable to connect to the source or target, it will eventually timeout and write a log showing the migration failed. However the dmt command will still be running with output like the following:

Hazelcast Data Migration Tool v5.3.0
(c) 2023 Hazelcast, Inc.

Selected data structures in the source cluster will be migrated to the target cluster.


    OK [1/2] Connected to the migration cluster.
    OK [2/2] Started the migration with ID: 078b5f14-aa73-4704-902d-a2c939397107.

Expected behavior: the dmt command should exit with some kind of message stating the migration cluster could not connect to either the source or target.

To reproduce:

1. From the DMT package directory, start the migration cluster w/ a command like: HZ_NETWORK_PORT_PORT=5702 HZ_CLUSTERNAME=migration ./bin/hz start (don't start any of the other clusters)
2. Start the migration: ./bin/dmt --config migration.yaml start migration_config --yes
3. Eventually a warning like the following will be logged by the migration cluster: 2023-11-06 13:42:03,357 [ WARN] [data-migration-command-queue-handler] [c.h.d.i.MigrationRunner]: [127.0.0.1]:5702 [migration] [5.3.5-DM-SNAPSHOT] Migration failed unexpectedly com.hazelcast.core.HazelcastException: Cannot connect to target cluster:Unable to connect to any cluster.
4. The dmt command will continue running, and no failure will be indicated to the user.

[github-actions]

Internal Jira issue: CLC-432

[sumnerib]

Similar behavior w/ the estimate command:

bin/dmt estimate --config migration.yaml migration_config

Hazelcast Data Migration Tool v5.3.0
(c) 2023 Hazelcast, Inc.


Estimation usually ends within 15 seconds.
    OK [1/2] Connected to the migration cluster.
       

[yuce]

Fixed by: #428