-1

I have about 1,000,000 records that I would like to loop through and write the contents in about 10 different files.

Below is my code that successfully create a single large file :

f = open("c:\temp\filename", 'w+')
# write to file
for rec in full_data:
    f.write(rec[0])
    f.write("\n")

I want to create 10 files with 100,000 records in each file.

I tried creating multiple files but this will create file for each row:

for index, line in enumerate(full_data):
    with open('filename{}.txt'.format(index), 'w') as output:
        output.write(line[0])

Please let me know how can create each file for 100,00 records.

martineau
  • 119,623
  • 25
  • 170
  • 301
labUser
  • 1

2 Answers2

1

That's because at each cicle iteration you are creating a new file as output, try to do that only when necessary.

for example, the following piece of code create a file every three items:

>>> data = [1,2,3,4,5,6,7,8,9,10]
>>> output = open('filename0.txt', 'w')
>>> for index, line in enumerate(data):
...     if index % 3 == 0:
...             output = open('filename{}.txt'.format(index), 'w')
...     output.write(str(line) + "\n")
tia.milani
  • 118
  • 1
  • 5
0

I was able to resolve this issue using below code:

        f = open(full_file_name, 'w+')
        rec_counter = 0
        file_name_counter = 1
        # write to file
        for index,rec in enumerate(full_run_data):
            if rec_counter == 500000: 
                f = open(repeat_file_name.format(file_name_counter), 'w')
                #print('%s' % (rec[0]))
                f.write(rec[0])
                f.write("\n")
                rec_counter = 0
                file_name_counter = file_name_counter + 1
            else:
                rec_counter = rec_counter +1
                f.write(rec[0])
                f.write("\n")
   
        f.close()
labUser
  • 1