Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retries when running git fetch #2405

Merged
merged 4 commits into from
Dec 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions build/scripts/EnlistmentHelperFunctions.psm1
Original file line number Diff line number Diff line change
Expand Up @@ -448,5 +448,74 @@
}
}

<#
.SYNOPSIS
Invokes a command with retry logic.
.DESCRIPTION
This function will invoke a command and retry it up to a specified number of times if it fails.
The function will sleep for an increasing amount of time between each retry.
The function will stop retrying if the maximum wait time is reached.
.PARAMETER ScriptBlock
The script block to invoke.
.PARAMETER RetryCount
The number of times to retry the command.
.PARAMETER MaxWaitTimeBeforeLastAttempt
The maximum time in seconds to wait before
.PARAMETER FirstDelay
The time in seconds to wait before the first retry.
.PARAMETER MaxWaitBetweenRetries
The maximum time in seconds to wait between retries.
#>
function Invoke-CommandWithRetry {
[CmdletBinding()]
param (
[parameter(Mandatory = $true)]
[System.Management.Automation.ScriptBlock] $ScriptBlock,
[parameter(Mandatory = $false)]
[int] $RetryCount = 3,
[parameter(Mandatory = $false)]
[int] $MaxWaitTimeBeforeLastAttempt = 2 * 60 * 60,
[parameter(Mandatory = $false)]
[int] $FirstDelay = 60,
[parameter(Mandatory = $false)]
[ValidateRange(0, 60 * 60)]
[int] $MaxWaitBetweenRetries = 60 * 60
)
# Initialize the variables that will tell us when we should stop trying
$startTime = Get-Date
$retryNo = 0
# Start trying...
$nextSleepTime = $FirstDelay
while ($true) {
$retryNo++
try {
Invoke-Command -ScriptBlock $ScriptBlock -OutVariable output | Out-Null
return $output # Success!
Dismissed Show dismissed Hide dismissed
}
catch [System.Exception] {
$exceptionMessage = $_.Exception.Message
$secondsSinceStart = ((Get-Date) - $startTime).TotalSeconds

# Determine if we should keep trying
$tryAgain = $retryNo -lt $RetryCount -and $secondsSinceStart -lt $MaxWaitTimeBeforeLastAttempt
# Try again, or stop?
if ($tryAgain) {
# Sleep
$sleepTime = [System.Math]::Min($nextSleepTime, $MaxWaitTimeBeforeLastAttempt - $secondsSinceStart) # don't sleep beyond the max time
$sleepTime = [System.Math]::Min($sleepTime, $MaxWaitBetweenRetries) # don't sleep for more than one hour (and don't go above what Start-Sleep can handle (2147483))
Write-Warning "Command failed with error '$exceptionMessage' in attempt no $retryNo after $secondsSinceStart seconds. Will retry up to $RetryCount times. Sleeping for $sleepTime seconds before trying again..."
Start-Sleep -Seconds $sleepTime
$nextSleepTime = 2 * $nextSleepTime # Next time sleep for longer
# Now try again
}
else {
# Failed!
$output | Write-Host
throw
}
}
}
}

Export-ModuleMember -Function *-*
Export-ModuleMember -Function RunAndCheck
6 changes: 4 additions & 2 deletions build/scripts/GuardingV2ExtensionsHelper.psm1
Original file line number Diff line number Diff line change
Expand Up @@ -258,8 +258,10 @@ function Test-IsStrictModeEnabled
}

function Get-MaxAllowedObsoleteVersion() {
git fetch origin main
$alGoSettings = $(git show origin/main:.github/AL-Go-Settings.json) | ConvertFrom-Json
Import-Module -Name $PSScriptRoot\EnlistmentHelperFunctions.psm1

Invoke-CommandWithRetry -ScriptBlock { RunAndCheck git fetch origin main }
$alGoSettings = $(RunAndCheck git show origin/main:.github/AL-Go-Settings.json) | ConvertFrom-Json
if (-not $alGoSettings.repoVersion) {
throw "Unable to find repoVersion in AL-Go-Settings.json"
}
Expand Down
Loading