Powershell read file in chunks

Question

I've had a script written in Powershell which transferred a file via FTP which worked fine by using:

$content = [System.IO.File]::ReadAllBytes($backup_app_data)

But this stopped working once the file size reached 2Gb, by throwing an error saying that this method is limited to reading files up to this size.

Exception calling "ReadAllBytes" with "1" argument(s): "The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

Now, I'm trying to modify my script and use an alternate approach with Read, which from what I understand will allow me to read the file in chunks and then write those chunks and so on.

Unfortunately although the script I've modified seems to be working, since a file is created on my FTP location, there is no data being transferred and no error is thrown by the script.

There is just a 0kb file created in the destination folder and the script ends.

I've tried to do some debugging, but I can't seem to find the issue. Below is the code I'm currently using.

$file = 'G:\Backups\DATA_backup_2016_09_05.zip'
$dest_name = 'DATA_backup_2016_09_05.zip'

$ftp = [System.Net.FtpWebRequest]::Create($ftp+$dest_name)
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)

$ftp.UseBinary = $true
$ftp.UsePassive = $true

# determine the size of the file
$file_size = (Get-Item $file).length
$chunk_size = 512mb

$bytes_to_read = $file_size
$iterations = $file_size / $chunk_size
$iter = 0

$fstream = [System.IO.FileStream]
[byte[]] $byte_array 

while ($bytes_to_read > 0){
    if($iterations > 1) {
        $content = $fstream.Read($byte_array, $iter, $chunk_size)
        }
    else {
        $content = $fstream.Read($byte_array, $iter, $bytes_to_read)
    }

    $ftp.ContentLength = $content.Length

    $rs = $ftp.GetRequestStream()
    $rs.Write($content, 0, $content.Length)

    # keep the loop going
    $iter = $iter + 1
    $iterations = $iterations - 1
    $bytes_to_read = $bytes_to_read - $chunk_size
}

$rs.Close()
$rs.Dispose()

Any help is appreciated.

Possible duplicate of [Upload BIG files via HTTP](http://stackoverflow.com/questions/33235806/upload-big-files-via-http) - While for HTTP, it uses the same API, the `WebRequest` and streams, so the loop from the [answer by the @vvchik](http://stackoverflow.com/a/33253233/850848) can be used for FTP too. — Martin Prikryl, Sep 06 '16 at 10:39
I suspect you need to get rid of your *iterations* variable, since in general you can't tell how many reads you are going to need up-front since you won't necessarily get as many bytes as you try to read. — Mark Setchell, Sep 06 '16 at 10:47
Thanks for the input guys, I'm currently working on a version of the script based on the suggested solution from the other question. I'll post my findings and hopefully a working script as soon as possible. — Radu Gheorghiu, Sep 06 '16 at 19:25

score 3 · Accepted Answer · edited Sep 12 '19 at 00:18

So, based on the suggestions in the comment section and using as an example the answer from the possible duplicate question, I managed to find the solution myself.

Below is the code I used to transfer via FTP a .zip archive which was 3.74Gb.

$ftp_addr = "ftp://ftp.somerandomwebsite.com/folder1/"
$user = "usr"
$pass = "passwrd"
$bufSize = 256mb

# ... missing code where I identify the file I want to FTP ... #

# Initialize connection to FTP
$ftp = [System.Net.FtpWebRequest]::Create($ftp_addr + $backup_file + ".zip")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)

$ftp.Timeout = -1              #infinite timeout
$ftp.ReadWriteTimeout = -1     #infinite timeout

$ftp.UseBinary = $true
$ftp.UsePassive = $true

$requestStream = $ftp.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file_to_ftp)
$chunk = New-Object byte[] $bufSize

while ( $bytesRead = $fileStream.Read($chunk, 0, $bufSize) ){
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
}

$fileStream.Close()
$requestStream.Close()

So far, this code has worked flawlessly for about multiple (20+) attempts. I hope it helps others!

Powershell read file in chunks

1 Answers1

Linked