I Have erratic file transfers with Powershell, FileSytemWatcher and WinSCP

Question

I have a problem with my PowerShell script putting files to an ESB. Often it works, but not always, especially with large files things go wrong. Does anyone see errors in my script or a way to tackle/resolve my problem?

Note:

Resume support is not an option because I have no change rights.
With "WinSCP GUI" I manage to successfully upload large files. (resume support off)
The directory which contains the files for upload, is monitored for file creation, then the files are copied to a separate folder for upload. When upload is successful, the file is deleted(from the second folder).
I'm a beginner at PowerShell, so there's probably the reason for error(s). English is also not my native language, so excuse me for any mistakes.

The script is started in TaskScheduler:

    Powershell.exe -NoExit -File "script.ps1"

Here's the script:

    $folder = 'Original_Folder'
    $filter = '*.xml'                             
    $destination = 'CopyTo_Folder'
    $fsw = New-Object IO.FileSystemWatcher $folder, $filter -Property @{
    IncludeSubdirectories = $false              
    NotifyFilter = [IO.NotifyFilters]'FileName, LastWrite'
    }
    $onCreated = Register-ObjectEvent $fsw Created -SourceIdentifier FileCreated -Action {
    $path = $Event.SourceEventArgs.FullPath
    $name = $Event.SourceEventArgs.Name
    $changeType = $Event.SourceEventArgs.ChangeType
    $timeStamp = $Event.TimeGenerated
    Write-Host "$timeStamp The file '$name' was $changeType and copied for SFTP to $destination"
    "$timeStamp The file '$name' was $changeType and copied for SFTP to $destination" | Set-Content 'Powershell_Script.log'
    Copy-Item $path -Destination $destination -Force -Verbose

    # Load WinSCP .NET assembly
    Add-Type -Path "C:\Program Files (x86)\WinSCP_5_11_3\WinSCPnet.dll"

    # Set up session options
    $sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Sftp
    HostName = "ftp-server"
    UserName = "User account"
    Password = "User password"
    SshHostKeyFingerprint = "ssh-rsa bla bla"
    }

    $sessionOptions.AddRawSettings("FSProtocol", "2")

    $session = New-Object WinSCP.Session
    $Session.SessionLogPath="WinSCP.log"
    try
    {
        # Connect
        $session.Open($sessionOptions)

        # Upload files

            # Set up transfer options
            $transferOptions = New-Object WinSCP.TransferOptions -Property @{
            ResumeSupport = New-Object WinSCP.TransferResumeSupport -Property @{ State = [WinSCP.TransferResumeSupportState]::Off }
            }

            # Upload with transferoptions and delete original when succesfull    
            $transferResult=$session.PutFiles("CopyTo_Folder\*", "/Upload_Folder on ftp-server/", $True, $transferOptions)

            # Throw on any error
            $transferResult.Check()

            # Print results
            foreach ($transfer in $transferResult.Transfers)
            {
                Write-Host "Upload of $($transfer.FileName) succeeded"
                "$timeStamp The file $($transfer.FileName) was successfully uploaded" | Add-Content 'Powershell_Script.log'
            }

    }
    finally
    {
        $session.Dispose()
    }
    }

What happens when there's an error? Partial file transfer? File is totally broken? Nothing happens? — vonPryz, Dec 16 '19 at 13:18
Thanks for answering vonPryz. Partial files are transfered sometimes and partial content is missing. Screen output says nothing as if it's still monitoring. By the way the script is started in as: Powershell.exe -NoExit -File "script.ps1" — d1gg0, Dec 16 '19 at 13:45
Show us a WinSCP session log file (`Session.SessionLogPath`) for a session that has the problem. — Martin Prikryl, Dec 16 '19 at 15:01
I Would love to, but when files are not completed, there is no log.. Script hangs i suppose. — d1gg0, Dec 16 '19 at 15:32
Is there's a log file, when the upload works? If there it, there must also be a log file when a part of the file is uploaded. -- The only case when there's no log would be if the script does not even get to the `Open` call - or when the logging never works. — Martin Prikryl, Dec 16 '19 at 15:38
As I use FileSystemWatcher and there are large files involved. I will try and put some Start-Sleep moments in. Maybe FSW is picking up files while being copied... — d1gg0, Dec 16 '19 at 15:42
Hi Martin, logging works when files are completed. I can't find logs of the partial files. — d1gg0, Dec 16 '19 at 15:45
Did you try creating your own log records in your PowerShell script? — Martin Prikryl, Dec 16 '19 at 16:30

score 0 · Accepted Answer · answered Dec 17 '19 at 13:33

I seem to have tackled the problem.

When getting the copy action in between pauses, it works fine

So:

    Start-Sleep -s 30
    Copy-Item $path -Destination $destination -Force -Verbose
    Start-Sleep -s 30

FileSystemWatcher acts inmediatly, as should be expected? So, the file is copied while it isn't totally created. After the copy command the upload will immediately start so I put a pause there too so the copy action can finish before upload starts.

It would be better to monitor these actions too and let the script resume after the actions are completed. Although I don't know if that is possible for the file creation with FSW...

I Have erratic file transfers with Powershell, FileSytemWatcher and WinSCP

1 Answers1