Extract data from csv file and save to text file (without overloading your RAM)

Question

I had this huge list of providers ip address ranges with additional info about the total number of addresses and the name of the provider.
It pretty much looks like this:

2.160.0.0,2.175.255.255,1048576,28/09/10,Telekom Deutschland GmbH
2.200.0.0,2.207.255.255,524288,18/11/10,Vodafone GmbH

To feed it to another program I had to convert it to a simple text file that contains a basic list of ip ranges. Like this:

2.160.0.0-2.175.255.255
2.200.0.0-2.207.255.255

So since the file is really huge the question would be:
How can I convert this kind of csv table into a txt based ip range list, without loading the whole file into the RAM at the same time?

Answer 1

Well, this is the answer I came up with. Pure AutoHotkey and only loads one line at a time into the RAM and it's really blazing fast and reliable.

;Set input and output files
inputFile := "log.csv"
outputFile := "out.txt"

;If the output file exists already, delete it before we start.
If FileExist(outputFile) { 
    FileDelete, %outputFile%
    If ErrorLevel
        MsgBox, Error: Can't access "%outputFile%"!
}

;Count the lines of our input file
Loop, Read, %inputFile%
    lineCount := A_Index

;Parse the input file, filter it and output it to the output file
Loop, Read, %inputFile%, %outputFile%
{
    currentLine := A_LoopReadLine
    If (currentLine != "") { ;If the current line is not empty
        currentLineArray := StrSplit(currentLine,",") ;Split the line by the comma
        stringToWrite := currentLineArray[1] "-" currentLineArray[2] . "`r`n" ;generate a basic ip range line with a dash as a seperator
        FileAppend, %stringToWrite% ;write it to our output file
        ;TrayTip, Writing to ip range file..., Line %A_Index%/%lineCount%
    }
}