Cryptography. English "normal text"?

Question

I was asked to make a software that will encrypt and decrypt a "normal English" text based on letter frequencies.

The question is where do I find some text samples where the official frequencies will match?

So far, I have tried "War and Peace" by Lev Tolstoy, it didn't work well..

LE: I don't need just a list of words, I need a text sample to make some processing.
LE2: The goal is to guess 20 from 26 in a 2000 characters long text.

IN, SHR, CUMW wrong, even on 2.6MB .txt file... I would expect wrong frequencies on smaller parts of the text.. but not on the whole book. — sdadffdfd, Oct 16 '10 at 03:49
If I cut the file to ~100KB I can match about 6-10 letters, no more. — sdadffdfd, Oct 16 '10 at 03:50
The goal is to guess 20 from 26 in a 2000 characters long text. — sdadffdfd, Oct 16 '10 at 03:57

score 2 · Answer 1 · answered Oct 16 '10 at 03:15

2

Check out infochimps; they have a bunch of freely available datasets that may be useful.

answered Oct 16 '10 at 03:15

Noon Silk

54,084
6
88
105

Darius Bacon · Accepted Answer · 2010-10-16T03:32:17.893

2

You're searching for English text corpora, e.g. http://faculty.washington.edu/ebender/corpora/corpora.html#modern. Out of what's listed there, I know that Project Gutenberg is free; many of the others might not be.

I'm not sure what you mean by the official frequencies -- the point of the frequencies is to match what you find in the wild, and if they don't, that's the frequency table's problem.

edited Oct 16 '10 at 03:32

answered Oct 16 '10 at 03:25

Darius Bacon

14,921
5
53
53

thanks, found some text samples, going to test frequencies now. – sdadffdfd Oct 16 '10 at 04:15

score 1 · Answer 3 · answered Oct 16 '10 at 03:49

1

Try this list of English words:

http://www.openbsd.org/cgi-bin/cvsweb/src/share/dict/

answered Oct 16 '10 at 03:49

Sean Comeau

11
1

Cryptography. English "normal text"?

3 Answers3