Below program, which extracts some specific column values from a text file delimited by separator "~" is throwing memory error when the number of records in the text file are around 6,000,000 PLUS. However the same snippet is working for smaller number of records in the text file. I tried using list comprehension but not able to resolve the memory error. Below is the code snippet:
import os
import csv
print("Analyzing the File!")
tnum = list()
nl = 0
total = list()
fdata = list()
amount = list()
with open(r"C:\Users\aojha\Desktop\20191125\ABC.TXT", "r") as fh:
next(fh)
for line in fh:
line = line.rstrip()
nl+=1
print("Number of records in file are:",nl)
with open(r"C:\Users\aojha\Desktop\20191125\ABC.TXT", "r") as fh:
next(fh)
fdata = [line.split("~") for line in fh]
##print(fdata)
for data in fdata:
tnum.append((data[2][:]).split(" "))
total.append((data[8][:]))
amount.append((data[13][:]))
#print(fdata)
print("Required data from file have been extracted!")}
Sample Input Data is in below format:
P~LNL~22248370~50~22248370~20190916~20191112~20190916~002~I~A~N~003~1638~01~001~400023~-1552~20190916~0200058~001~X~~TMID~~~~~000~000~~000~000~000~~000~000~0~~~~N~~~
Error received is as follows:
> Traceback (most recent call last):
File "C:\Users\aojha\Desktop\Python\Random\Type_Count.py", line 23, in <module>
fdata = [line.split("~") for line in fh]
File "C:\Users\aojha\Desktop\Python\Random\Type_Count.py", line 23, in <listcomp>
fdata = [line.split("~") for line in fh]
MemoryError
