27

I have a fairly large powershell scripts with many (20+) functions which perform various actions.

Right now all of the code doesn't really have any error handling or retry functionality. If a particular task/function fails it just fails and continues on.

I would like to improve error handling and implement retries to make it more robust.

I was thinking something similar to this:

$tries = 0
while ($tries -lt 5) {
    try{    

       # Do Something

       # No retries necessary
       $tries = 5;
    } catch {
       # Report the error
       # Other error handling
    }
 }

The problem is that I have many many steps where I would need to do this.

I don't think it make sense to implement the above code 20 times. That seems really superfluous.

I was thinking about writing an "TryCatch" function with a single parameter that contains the actual function I want to call?

I'm not sure that's the right approach either though. Won't I end up with a script that reads something like:

TryCatch "Function1 Parameter1 Parameter2"
TryCatch "Function2 Parameter1 Parameter2"
TryCatch "Function3 Parameter1 Parameter2"

Is there a better way to do this?

Mathias R. Jessen
  • 157,619
  • 12
  • 148
  • 206
Brad
  • 1,979
  • 7
  • 35
  • 47

4 Answers4

55

If you frequently need code that retries an action a number of times you could wrap your looped try..catch in a function and pass the command in a scriptblock:

function Retry-Command {
    [CmdletBinding()]
    Param(
        [Parameter(Position=0, Mandatory=$true)]
        [scriptblock]$ScriptBlock,

        [Parameter(Position=1, Mandatory=$false)]
        [int]$Maximum = 5,

        [Parameter(Position=2, Mandatory=$false)]
        [int]$Delay = 100
    )

    Begin {
        $cnt = 0
    }

    Process {
        do {
            $cnt++
            try {
                # If you want messages from the ScriptBlock
                # Invoke-Command -Command $ScriptBlock
                # Otherwise use this command which won't display underlying script messages
                $ScriptBlock.Invoke()
                return
            } catch {
                Write-Error $_.Exception.InnerException.Message -ErrorAction Continue
                Start-Sleep -Milliseconds $Delay
            }
        } while ($cnt -lt $Maximum)

        # Throw an error after $Maximum unsuccessful invocations. Doesn't need
        # a condition, since the function returns upon successful invocation.
        throw 'Execution failed.'
    }
}

Invoke the function like this (default is 5 retries):

Retry-Command -ScriptBlock {
    # do something
}

or like this (if you need a different amount of retries in some cases):

Retry-Command -ScriptBlock {
    # do something
} -Maximum 10

The function could be further improved e.g. by making script termination after $Maximum failed attempts configurable with another parameter, so that you can have have actions that will cause the script to stop when they fail, as well as actions whose failures can be ignored.

dazbradbury
  • 5,729
  • 5
  • 34
  • 38
Ansgar Wiechers
  • 193,178
  • 25
  • 254
  • 328
  • Tested this out and it seems as though with $ScriptBlock.Invoke() if there's an error its not being caught. With write-host statements I confirmed that retry-command is being called, make it into the do loop, make it into the try loop, but i've intentionally entered data into my scriptblock which would cause an error yet no error is caught – Brad Aug 03 '17 at 11:57
  • My script block looks like this: Invoke-Sqlcmd -Query "DELETE FROM This_Table_Does_NOT_Exist" -ServerInstance mydbserver -Database TestDB. If I run the command from outside my script I do get an error: Invoke-Sqlcmd : Invalid object name 'This_Table_Does_NOT_Exist'. – Brad Aug 03 '17 at 11:59
  • The error appears to be a [non-terminating error](https://blogs.technet.microsoft.com/heyscriptingguy/2015/09/16/understanding-non-terminating-errors-in-powershell/). Try adding `-ErrorAction Stop` to the `Invoke-Sqlcmd` statement. – Ansgar Wiechers Aug 03 '17 at 12:01
  • I think there's another problem which is that the variables aren't being interpreted. I tried adding Param($ServerName,$SQLInstance, $DatabaseName) to the script block code but it seems like they're still not being interpreted. – Brad Aug 03 '17 at 12:47
  • 1
    I tested the code before posting, and variables could be used just fine. My sample code was missing a `-ErrorAction Continue` for the `Write-Error` statement, though (so that the code won't terminate at the first error). – Ansgar Wiechers Aug 03 '17 at 13:08
  • Sorry you're right. It appeared the variables weren't being interpreted based on the output but with the ErrorAction set correctly in various places it DOES work at expected. Just very misleading how the error was output. Thanks Again. – Brad Aug 03 '17 at 15:55
  • 1
    I recommend a configurable sleep after writing out the exception. – CrazyIvan1974 Nov 07 '19 at 15:45
  • 1
    If one wants to see the success and verbose stream of the $ScriptBlock in case of exception, `Invoke-Command -Command $ScriptBlock` may be better than `$ScriptBlock.Invoke()`. It seems that `$ScriptBlock.Invoke()` returns those streams and they don't get displayed when the script block throws an exception. In this case, use `Write-Error $_.Exception.Message -ErrorAction Continue` to display the exception's message. – GeorgesZ Jul 20 '21 at 12:51
  • I recommend a `-On { }` parameter which only reties if the condition is met and otherwise (re)throws the current error. – iRon Oct 04 '22 at 06:30
  • One more recommendation - you should make sure you specify the -Message property for the Write-Error otherwise the -ErrorAction doesn't take affect: `Write-Error -Message $_.Exception.InnerException.Message -ErrorAction Continue` – Jack S Mar 03 '23 at 11:50
10

I adapted @Victor's answer and added:

  • parameter for retries
  • ErrorAction set and restore (or else exceptions do not get caught)
  • exponential backoff delay (I know the OP didn't ask for this, but I use it)
  • got rid of VSCode warnings (i.e. replaced sleep with Start-Sleep)
# [Solution with passing a delegate into a function instead of script block](https://stackoverflow.com/a/47712807/)
function Retry()
{
    param(
        [Parameter(Mandatory=$true)][Action]$action,
        [Parameter(Mandatory=$false)][int]$maxAttempts = 3
    )

    $attempts=1    
    $ErrorActionPreferenceToRestore = $ErrorActionPreference
    $ErrorActionPreference = "Stop"

    do
    {
        try
        {
            $action.Invoke();
            break;
        }
        catch [Exception]
        {
            Write-Host $_.Exception.Message
        }

        # exponential backoff delay
        $attempts++
        if ($attempts -le $maxAttempts) {
            $retryDelaySeconds = [math]::Pow(2, $attempts)
            $retryDelaySeconds = $retryDelaySeconds - 1  # Exponential Backoff Max == (2^n)-1
            Write-Host("Action failed. Waiting " + $retryDelaySeconds + " seconds before attempt " + $attempts + " of " + $maxAttempts + ".")
            Start-Sleep $retryDelaySeconds 
        }
        else {
            $ErrorActionPreference = $ErrorActionPreferenceToRestore
            Write-Error $_.Exception.Message
        }
    } while ($attempts -le $maxAttempts)
    $ErrorActionPreference = $ErrorActionPreferenceToRestore
}

# function MyFunction($inputArg)
# {
#     Throw $inputArg
# }

# #Example of a call:
# Retry({MyFunction "Oh no! It happened again!"})
# Retry {MyFunction "Oh no! It happened again!"} -maxAttempts 10
Colin
  • 1,987
  • 3
  • 17
  • 21
7

Solution with passing a delegate into a function instead of script block:

function Retry([Action]$action)
{
    $attempts=3    
    $sleepInSeconds=5
    do
    {
        try
        {
            $action.Invoke();
            break;
        }
        catch [Exception]
        {
            Write-Host $_.Exception.Message
        }            
        $attempts--
        if ($attempts -gt 0) { sleep $sleepInSeconds }
    } while ($attempts -gt 0)    
}

function MyFunction($inputArg)
{
    Throw $inputArg
}

#Example of a call:
Retry({MyFunction "Oh no! It happend again!"})
Victor Fialkin
  • 165
  • 4
  • 7
0

Error handling is always going to add more to your script since it usually has to handle many different things. A Try Catch function would probably work best for what you are describing above if you want to have each function have multiple tries. A custom function would allow you to even set things like a sleep timer between tries by passing in a value each time, or to vary how many tries the function will attempt.

Jason Snell
  • 1,425
  • 12
  • 22
  • You may also want to figure out why you need to retry steps 5 times. Anyway, having a trycatch function would also be a good place to create log output and maybe sort that out... – brendan62269 Aug 03 '17 at 03:54
  • Its mostly a precaution. So there are a few error scenarios we're trying to cover: 1) Is this a terminal error (should we exit) 2) Is this a non-terminal error and we should continue but warn about the error 3) Is this a temporary error (will retrying help?). An example may be that we're trying to run an operation against a VM or cloud instance which hasn't fully started yet. Waiting and retrying is a good option in this situation. Although eventually it should give up. – Brad Aug 03 '17 at 12:59