Editing large text files on linux ( 5 - 10gb)

Question

Basically, i need a file of specified format and large size(Around 10gb). To get this, i am copying the contents of my original file into the same file, multiple times, to increase its size. I dont care about the contents of the file as long as they have the required format. Initially, i tried to do this using gedit, which failed miserably after few 100mbs. I'm looking for an editor which will help me do this. Or, may be a suggestion on alternate ways

Are you sure you need to manually edit such a file? What could a 10GB file be, but perhaps a log or a flat-file "database"? Generally things like this would be "edited" programmatically, I think. — Ray Toal, Jul 26 '11 at 00:55
I wanted a sample file to see how my application responded to files of extremely-large-size — Chander Shivdasani, Jul 27 '11 at 00:49
Ah, I see, that's totally cool. Jason has a good answer below. — Ray Toal, Jul 27 '11 at 00:56

score 2 · Answer 1 · answered Jul 26 '11 at 00:57

2

You could make 2 files and repeatedly append them to each other:

cp file1 file2

for x in `seq 1 200`; do 
       cat file1 >> file2
       cat file2 >> file1
done;

answered Jul 26 '11 at 00:57

Dave

10,964
3
32
54

score 1 · Accepted Answer · answered Jul 26 '11 at 01:21

In Windows, from the command line:

copy file1.txt+file2.txt file3.txt

concats 1 and 2, places in 3 - repeat or add +args until you get the size you need.

For Unix,

cat file1.txt file2.txt >> file3.txt

concats 1 and 2, places in 3 - repeat or add more input files until you get the size you need.

There are probably many other ways to do this in Unix.

Editing large text files on linux ( 5 - 10gb)

2 Answers2