Reliable file transfer between three systems

Question

I've got three systems I need to transfer files between daily:

Vendor A (SFTP site) - Deletes files after 14 days
Vendor B (SFTP site) - Deletes files after processing them
Company Server (Windows 2008) - Stores files as backups

Each day I want to download new files from Vendor A to our company server and upload them to Vendor B. The caveat with a simple file transfer script is that the file transfer from Vendor A to the company server might succeed but fail when transferring from company server to Vendor B. Thus vendor B is missing a days worth of files. Vendor B doesn't tell me when they are missing files and Vendor A doesn't always generate a file on a given day.

Any ideas on how to automate this file transfer scenario so that in the event of a failure the system will try again at a later time?

score 1 · Answer 1 · answered Aug 29 '14 at 12:32

You just have to add some logic to retry the upload to Vendor B after some period of time. In case you reach the maximum number of failures, log an error or send an email to have it verified manually by the support team.

If you expect Vendor A won't generate files every day, determine how important the missing days are the decide if you should also generate an alert/email for those or not (e.g. you have to call Vendor A to check why the file is missing at the expected time).

Add in *serious* logging, and alerting as appropriate. – mfinni Aug 29 '14 at 12:41 — mfinni, Aug 29 '14 at 12:41

Martin Prikryl · Answer 2 · 2017-03-08T13:03:31.150

I'd do:

One-way synchronization of files (Vendor A -> Company Server "backup" folder).
Clone newly downloaded files to a separate folder with files to upload ("upload" folder).
Check how old the latest file in "backup" folder is. When older than X days, alert.
Try to upload all (not only new) files from "upload" folder to Vendor B
Remove files that you succeed to upload from "upload" folder
Alert on any (download or upload) error

Repeat this at least once a day, possibly few times a day. Or do the steps 1-3 once a day, but the steps 4-6 few times a day.

Possible implementation using PowerShell:

Use the WinSCP .NET assembly method Session.SynchronizeDirectories:

$synchronizationResult = $session.SynchronizeDirectories(
    [WinSCP.SynchronizationMode]::Local, "c:\backup\", "/remote_path/", $False)

Iterate the SynchronizationResult.Downloads and Copy-Item each:

foreach ($download in $synchronizationResult.Downloads)
{
    Copy-Item $download.Destination -Destination "C:\upload"
}

Use the Get-ChildItem to find the latest file:

# Find the latest file
$latest = Get-ChildItem "C:\backup" |
    Sort-Object LastWriteTime -Descending | Select-Object -First 1

# Is the latest file older than 5 days?
if ($latest.LastWriteTime -lt (Get-Date).AddDays(-5))
{
    # alert!
}

Use the WinSCP .NET assembly method Session.PutFiles:

$transferResult = $session.PutFiles("C:\upload\*.*", "/remote_path/")

Iterate TransferResult.Transfers and Remove-Item each:

foreach ($transfer in $transferResult.Transfers)
{
    if ($transfer.Error -eq $Null)
    {
        Remove-Item $transfer.FileName
    }
}

Test both the SynchronizationResult.IsSuccess and the TransferResult.IsSuccess

(I'm the author of WinSCP)

Reliable file transfer between three systems

2 Answers2