Skip to content

add check before fencing #10059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: 4.19
Choose a base branch
from
Draft

add check before fencing #10059

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3ef3025
add check befor fencing
leo79901 Dec 4, 2024
f5dee3b
rename logger to prevent merge forward conflict
DaanHoogland Dec 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,13 @@
import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
import org.apache.log4j.Logger;
import org.joda.time.DateTime;
import com.cloud.host.Status;

import javax.inject.Inject;
import java.security.InvalidParameterException;

public final class KVMHAProvider extends HAAbstractHostProvider implements HAProvider<Host>, Configurable {
private final static Logger LOG = Logger.getLogger(KVMHAProvider.class);
private final static Logger logger = Logger.getLogger(KVMHAProvider.class);

@Inject
protected KVMHostActivityChecker hostActivityChecker;
Expand Down Expand Up @@ -75,28 +76,34 @@
final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.RESET, null);
return resp.getSuccess();
} else {
LOG.warn("OOBM recover operation failed for the host " + r.getName());
logger.warn("OOBM recover operation failed for the host " + r.getName());

Check warning on line 79 in plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

View check run for this annotation

Codecov / codecov/patch

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java#L79 

Added line #L79 was not covered by tests
return false;
}
} catch (Exception e){
LOG.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());
logger.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());

Check warning on line 83 in plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

View check run for this annotation

Codecov / codecov/patch

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java#L83 

Added line #L83 was not covered by tests
throw new HARecoveryException(" OOBM service is not configured or enabled for this host " + r.getName(), e);
}
}

@Override
public boolean fence(Host r) throws HAFenceException {

try {
if (outOfBandManagementService.isOutOfBandManagementEnabled(r)){
final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
return resp.getSuccess();
// host exists and is managed OOB
if (r != null && outOfBandManagementService.isOutOfBandManagementEnabled(r)) {
// check host status
if (Status.Down.equals(r.getStatus())) {
Copy link
Contributor

@slavkap slavkap Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DaanHoogland, my concern here is that host's status could be Down but in reality to be still running. With this there is an option the virtual machines started to a neighbor host to be left running on the previous one.
Maybe it is better to check the OOBM status rather the host's.
Unfortunately now I don't have environment with IPMI setup and cannot test this change.
Probably here and @rp- could give some advice.

Copy link
Contributor

@rp- rp- Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True what @slavkap said, and now I'm a bit confused what actual the original issue was?
Wouldn't it be not just a noop if the host is already down and fence it again?

Copy link
Contributor Author

@DaanHoogland DaanHoogland Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean to change this condition to something else , @slavkap ?
If it is !Down a power operation will be performed. should it happen here as well?
I am not sure if we have a OOB status retrieval possibility.

Copy link
Contributor

@slavkap slavkap Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rp-, there was a discussion about an issue with the fencing that the Host is Powered off and CS tries to shut it down again. But I've seen cases where the host could be in the down state before the IPMI tries to power it off.
@DaanHoogland, yes, I think there is an option to check the OOBM status and the condition should be changed with the result of it

Copy link
Contributor Author

@DaanHoogland DaanHoogland Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, it is not part of outOfBandManagementService.

logger.info("Host " + r.getName() + " is already down. Returning success.");
return true;

Check warning on line 96 in plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

View check run for this annotation

Codecov / codecov/patch

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java#L95-L96 

Added lines #L95 - L96 were not covered by tests
} else {
final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
return resp.getSuccess();

Check warning on line 99 in plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

View check run for this annotation

Codecov / codecov/patch

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java#L98-L99 

Added lines #L98 - L99 were not covered by tests
}
} else {
LOG.warn("OOBM fence operation failed for this host " + r.getName());
logger.warn("OOBM fence operation failed for this host " + r.getName());

Check warning on line 102 in plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

View check run for this annotation

Codecov / codecov/patch

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java#L102 

Added line #L102 was not covered by tests
return false;
}
} catch (Exception e){
LOG.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());
logger.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());

Check warning on line 106 in plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

View check run for this annotation

Codecov / codecov/patch

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java#L106 

Added line #L106 was not covered by tests
throw new HAFenceException("OBM service is not configured or enabled for this host " + r.getName() , e);
}
}
Expand Down
Loading