0

Using VBScript on local Windows 10 machine to automate some tasks. Dealing with large binary files (500 MB up to 2 GB). FileSystemObject's file handling is abysmally slow! ADODB is fast, but loads the entire file into memory, a big no-no since I will be running the script on multiple files (on multiple disks) concurrently. Any alternatives?

Below is an example of FSO taking about 26 seconds to handle a 607 MBs file. Skipping 600 MBs alone consumes a staggering 25 seconds!

Dim i, f1, f2, dT
Dim objFSO1, objStream1, objFSO2, objStream2

f1 = "OriginalFile.tmp"
f2 = "OutputFile.tmp"

Set objFSO1 = CreateObject("Scripting.FileSystemObject")
Set objStream1 = objFSO1.OpenTextFile(f1, 1)

Set objFSO2 = CreateObject("Scripting.FileSystemObject")
Set objStream2 = objFSO2.OpenTextFile(f2, 8, True)

dT = Timer
For i = 0 To 4
    'Call objStream2.Write(objStream1.Read(1048576))
    Call objStream1.Read(1048576)
Next
dT = Timer - dT
Call WScript.Echo(dT & " seconds so far...")

dT = Timer
Call objStream1.Skip(629145600)
dT = Timer - dT
Call WScript.Echo("Skipping took " & dT & " seconds!")

dT = Timer
Do While Not objStream1.AtEndOfStream
    'Call objStream2.Write(objStream1.Read(1048576))
    Call objStream1.Read(1048576)
Loop
dT = Timer - dT
Call WScript.Echo("Another " & dT & " seconds.")

Call objStream1.Close(): Call objStream2.Close()
Set objStream1 = Nothing: Set objStream2 = Nothing
Set objFSO1 = Nothing: Set objFSO2 = Nothing

As you can see, testing while only reading the data off the disk to eliminate any possibility that the slow-down is caused by writing to disk. Same difference!

Makaveli84
  • 453
  • 6
  • 16
  • Copy the file locally and test that - odds are reading across the network is slowing things down even further. – Dave Feb 17 '18 at 19:47
  • I am reading locally. – Makaveli84 Feb 17 '18 at 20:00
  • Try `SAPI.spFileStream` or `MSXML2.XMLHTTP` to read the content of the local file as binary. Note `spFileStream` allows to load the content by chunks and reduce memory usage. – omegastripes Feb 18 '18 at 19:10
  • @omegastripes Thanks for the suggestion. I have opted to use Sapien's `ActiveXPoSh` PowerShell `COM` interface since PowerShell opens up access to the whole .NET `System.IO` library (which is much more flexible than all other alternatives). – Makaveli84 Feb 21 '18 at 11:22
  • On a separate level -- and this is possibly more `SuperUser` than `StackOverflow` -- the original issue at hand (the awful slowness of reading/seeking with `FSO`) is still puzzling me. From my tests, it is clear that `FSO` is reading/seeking at a very primitive (and possibly "software") level, i.e. byte-by-byte. I'm not sure if that's by design or not, but It seems that `FSO` is failing to invoke the system's IO API (and by extension the faster "hardware drivers" access? Just speculating here...) – Makaveli84 Feb 21 '18 at 11:30

0 Answers0