Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correctly Update Job Status to Running After Agent Restarts #386

Closed
wants to merge 10 commits into from
Closed

Correctly Update Job Status to Running After Agent Restarts #386

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c2cbda1
updates job status to Running on agent relaunch (instead of Starting)
Mar 14, 2025
825c946
testing job status update
Mar 14, 2025
a00454f
correctly updates job status once all agents checked in - fixed test …
Mar 14, 2025
94c4bb8
test unblocked
Mar 14, 2025
12b79a8
added null check
Mar 18, 2025
1735326
move update job logic to listener, let agentwatchdog to only handle a…
Mar 26, 2025
83290be
listener is not the key - updated the dreaded VMTrackerImpl job statu…
Mar 26, 2025
8a9486c
this should work for testing relaunches
Mar 27, 2025
dfe62e7
added try catch
Mar 27, 2025
0ec23dd
not working to simulate agent failure
Mar 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions tank_vmManager/src/main/java/com/intuit/tank/vmManager/VMTrackerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,11 +312,19 @@ private void addStatusToJobContainer(CloudVmStatus status, CloudVmStatusContaine
boolean rampPaused = true;
boolean stopped = true;
boolean running = true;
int activeAgentCount = 0;

LOG.debug(new ObjectMessage(ImmutableMap.of("jobId", cloudVmStatusContainer.getJobId(), "message", "DEBUG: Starting aggregation loop", "totalStatuses", cloudVmStatusContainer.getStatuses().size())));

// look up the job
JobInstance job = jobInstanceDao.get().findById(Integer.parseInt(status.getJobId()));
for (CloudVmStatus s : cloudVmStatusContainer.getStatuses()) {
JobStatus jobStatus = s.getJobStatus();
boolean isActive = (s.getEndTime() == null && s.getVmStatus() != VMStatus.terminated);
if (isActive) {
activeAgentCount++; // track active agents (tracking agent restarts)
}

if (jobStatus != JobStatus.Completed) { // If no VMs are Completed
isFinished = false;
}
Expand All @@ -329,10 +337,23 @@ private void addStatusToJobContainer(CloudVmStatus status, CloudVmStatusContaine
if (jobStatus != JobStatus.Stopped) { // If no VMs are Stopped
stopped = false;
}
if (jobStatus != JobStatus.Running) { // If no VMs are Running
running = false;
if (jobStatus != JobStatus.Running) {
if (isActive) {
running = false;
LOG.debug(new ObjectMessage(ImmutableMap.of("jobId", cloudVmStatusContainer.getJobId(), "message", "DEBUG: Active agent not Running, setting overall running=false", "instanceId", s.getInstanceId(), "agentJobStatus", jobStatus)));
} else {
// ignore terminated agent for running flag
LOG.debug(new ObjectMessage(ImmutableMap.of("jobId", cloudVmStatusContainer.getJobId(), "message", "DEBUG: Ignoring terminated agent status for 'running' flag check", "instanceId", s.getInstanceId(), "agentJobStatus", jobStatus)));
}
}
}

if (activeAgentCount == 0) {
// correcting running flag for zero active agents
LOG.debug(new ObjectMessage(ImmutableMap.of("jobId", cloudVmStatusContainer.getJobId(), "message", "DEBUG: No active agents found after loop, forcing running=false")));
running = false;
}

if (isFinished) {
LOG.info(new ObjectMessage(ImmutableMap.of("Message","Setting end time on container " + cloudVmStatusContainer.getJobId())));
if (cloudVmStatusContainer.getEndTime() == null) {
Expand Down