Powershell seems to treat sub pipeline code as part of execution body of first pipeline script block

Question

I am creating a retry command in powershell to invoke some script block with retry logic:

function Invoke-CommandWithRetry
{
    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [ScriptBlock]$Command,

        [Parameter(Mandatory = $false)]
        [ValidateRange(0, [UInt32]::MaxValue)]
        [UInt32]$Retry = 3,

        [Parameter(Mandatory = $false, Position = 3)]
        [ValidateRange(0, [UInt32]::MaxValue)]
        [UInt32]$DelayInMs = 1000
    )

    process {

        $retryCount = 0
        while( $true )
        {
            try 
            {
                return (Invoke-Command -ScriptBlock $Command)
            }
            catch
            {
                if( $retryCount -ge $Retry )
                {
                    throw
                }
                else
                {
                    Write-Warning $_.Exception
                    Start-Sleep -Milliseconds $DelayInMs
                    $retryCount++
                    Write-Warning "Retry command $retryCount time: $Command"
                }
            }
        }
    }
}

The command should be straightforward, try to run a script block and if exception found, re-run it.

But I found some issue with this function when the sub pipeline throw exception, here is my test code:

{
    Write-Host "Start";
    1,2,3 
} | Invoke-CommandWithRetry | ForEach-Object {
        $_
        throw "Error"
}
Write-Host "End"

I would expect that, because of exception, so 2,3 object will be skipped. The result is just as I expected, but the retry is working not only for the first pipeline script block, but seems also involve the following pipeline as the body of first pipeline, which is really weird for me, here is the result:

Start
1
WARNING: System.Management.Automation.RuntimeException: Error
WARNING: Retry command 1 time: 
    Write-Host "Start";
    1,2,3 

Start
1
WARNING: System.Management.Automation.RuntimeException: Error
WARNING: Retry command 2 time: 
    Write-Host "Start";
    1,2,3 

Start
1
WARNING: System.Management.Automation.RuntimeException: Error
WARNING: Retry command 3 time: 
    Write-Host "Start";
    1,2,3 

Start
1
Error
At line:7 char:9
+         throw "Error"
+         ~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Error:String) [], RuntimeException
    + FullyQualifiedErrorId : Error

Anyone can help to explain why this happened? and How to avoid the first script block retry when sub pipeline throw exception?