Looking for large text files for testing compression in all sizes

Question

I am looking for large text files for testing the compression and decompression in all sizes from 1kb to 100mb. Can someone please refer me to download it from some link ?

Answer 1

*** Linux users only ***

Arbitrarily large text files can be generated on Linux with the following command:

tr -dc "A-Za-z 0-9" < /dev/urandom | fold -w100|head -n 100000 > bigfile.txt

This command will generate a text file that will contain 100,000 lines of random text and look like this:

NsQlhbisDW5JVlLSaZVtCLSUUrkBijbkc5f9gFFscDkoGnN0J6GgIFqdCLyhbdWLHxRVY8IwDCrWF555JeY0yD0GtgH21NotZAEe
iWJR1A4 bxqq9VKKAzMJ0tW7TCOqNtMzVtPB6NrtCIg8NSmhrO7QjNcOzi4N b VGc0HB5HMNXdyEoWroU464ChM5R Lqdsm3iPo
1mz0cPKqobhjDYkvRs5LZO8n92GxEKGeCtt oX53Qu6T7O2E9nJLKoUeJI6Ul7keLsNGI2BC55qs7fhqW8eFDsGsLPaImF7kFJiz
...
...

On my Ubuntu 18 its size it about 10MB. Bumping up the number of lines, and thereby bumping up the size, is easy. Just increase the head -n 100000 part. So, say, this command:

tr -dc "A-Za-z 0-9" < /dev/urandom | fold -w100|head -n 1000000 > bigfile.txt

will generate a file with 1,000,000 of random lines of text and be around 100MB. On my commodity hardware the latter command takes about 3 seconds to finish.

Answer 2

And don't forget the collection of Corpus

The Canterbury Corpus
The Artificial Corpus
The Large Corpus
The Miscellaneous Corpus
The Calgary Corpus
The Canterbury Corpus

SEE: https://corpus.canterbury.ac.nz/descriptions/

there is a download links for the files available for each set

Answer 3

You can download enwik8 and enwik9 from here. They are respectively 100,000,000 and 1,000,000,000 bytes of text for compression benchmarks. You can always pull subsets of those for smaller tests.

Answer 4

Project Gutenberg looks exceptionally promising for this purpose. This resource contains thousands of books in many formats. Here is a sample of what is available:

Clicking on any of the links will reveal the the various formats that always include Plain Text UTF-8 and .txt:

Another possible place to get large amounts of random text data for compression testing would be data dump sites such as Wiki or even Stack Exchange

I've alse found this blog post that lists 10 open source places to get complex text data for analytics testing.

There are a lot of online resources for creating large text files of arbitrary or specific sizes, such as this Lorem Ipsum Generator which I have used for script development, but I've learned that these sources are no good for compression testing because the words tend to be limited and repetitive. Consequently, these files will compress considerably more than natural text would.

Answer 5

You can use this with python (download at python.org if you haven't already); this will generate a file fully made up of 'M's:

size = ''

while not size.isnumeric():
    size = input('How big would you like your file to be (mb)? ')

size = int(size)

name = input('Where would you like to locate the file? ')

open(name, 'w').write('')

for mb in range(size):
    open(name, 'a').write('M' * 1000000)

print('Done!')

Or, you can use this to generate a completely random file:

import random

size = ''
characters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"

while not size.isnumeric():
    size = input('How big would you like your file to be (mb)? ')

size = int(size)

name = input('Where would you like to locate the file? ')

open(name, 'w').write('')

for mb in range(size):
    open(name, 'a').write(''.join(random.choices(characters, k=1000000)))

print('Done!')

And finally, to generate a bunch of words with spaces between them:

import random

size = ''

words = '''
hello
goodbye
yay!
'''  # Your words list here, separated by newlines

words = words.split('\n')

wl = []

print('Interpreting list...\n')

for word in words:
    if word.strip():
        wl.append(word.strip() + ' ')
    words.remove(word)

while not size.isnumeric():
    size = input('How big would you like your file to be (\'1\' would be 1000 words)? ')

size = int(size)

name = input('What would you like to be the path to the file? ')

open(name, 'w').write('')

for word in range(size):
    open(name, 'a').write(''.join(random.choices(wl, k=1000)))

print('Done!')

Remember that some of these take longer than others; the wordslist one can take quite a while.