List store large amounts of data , Not enough memory

Question

public class ListKeywords
{
    public int ID { set; get; }
    public string Keyword { set; get; } //关键词
    public string Language { set; get; } //语种
    public int WordCount { set; get; } //单词数
    public int WordLength { set; get; } // 字符数
    public int Status { set; get; } //采集状态  0-未采集 1-采集成功 2-保存失败 3-保存成功 4-发布失败 5-发布成功
    public bool Taken { set; get; }
    public bool FTPStatus { set; get; }
    public bool DBStatus { set; get; }
    public string UrlName { set; get; }
    public ListKeywords()
    {
    }
    public ListKeywords(string keyword)
    {
        this.Keyword = keyword;
    }
}


List<string> lines = new List<string>();
List<ListKeywords> keywordsList = new List<ListKeywords>();

using (StreamReader sr = File.OpenText(filePath))
{
    string s = String.Empty;
    while ((s = sr.ReadLine()) != null)
    {
        //lines.Add(s);   //Operating normally
        eywordsList.Add(new ListKeywords("some keywords")); //  Operating normally
        keywordsList.Add(new ListKeywords(s)); // it will be out of memeory
    }
}

In text file, have 1,000,000 line data, if i use above code to load the large data to list< keywordsList >, it will raises an OutOfMemoryException, but if i load it to list< string >, it run normally. How to solved it ?

That's because your `ListKeywords` class consumes more memory than an empty string. Quite a bit more, actually. — Robert Harvey, Apr 15 '14 at 01:11
How long are these keywords on average? If they already consume a lot of memory the extra ~100 bytes (rough estimate) from each class instance might push it over the edge. (Also protip: there's no need to initialize s to String.Empty. Just leave it uninitialized.) — Matti Virkkunen, Apr 15 '14 at 01:14
It would really help if you explained what you are trying to accomplish, not just [what you are doing](http://meta.stackexchange.com/questions/66377/). It is pointless to open a file as a stream if you are just going to save everything. Why are you saving everything anyway? That `ListKeywords` doesn't look like it is doing anything with the data. Why do you have to save it? Can't you just save positions in the file or something? — Dour High Arch, Apr 15 '14 at 01:15
On another note, if you have 1,000,000 `ListKeywords` in memory, a `List` is probably not an efficient data structure to use. Depending on how you will access the data, a `Dictionary` may be appropriate where `T` is the data type used to locate an entry (could be the ID or the Keyword perhaps). — Eric J., Apr 15 '14 at 01:19
keywordsList.Add(new ListKeywords("keywords test")); this is also run normally. i want load keywords from txt file and process it, such as detected keywords language. — user1908725, Apr 15 '14 at 01:27
@Dour High Arch i need a array or list to store keywords, these keywords have: language,word count, status(process or not process),.... so i create a class to store my data. — user1908725, Apr 15 '14 at 01:30
Try running this in the Visual Studio Memory Profiler to understand exactly where memory is being allocated http://blogs.msdn.com/b/dotnet/archive/2013/04/04/net-memory-allocation-profiling-with-visual-studio-2012.aspx — Eric J., Apr 15 '14 at 01:34
I do not understand what “detected keywords language” means or why you think “a array or list to store keywords” will do that, but I assure you what you have posted will accomplish nothing whatsoever. You must include much more detail in your question. — Dour High Arch, Apr 15 '14 at 01:35
@Dour High Arch I want deal with txt file there are have a lot of words, on per line. I import it to memory and process it, such detect language each keyword, then i can export these results to txt files. I want to store in memory because I think it will faster. — user1908725, Apr 15 '14 at 01:45

Jordan · Accepted Answer · 2014-08-26T17:43:44.650

Instead of using a List maybe try using an IEnumerable w/ yield?

static IEnumerable<ListKeywords> Keywords()
{
    using (StreamReader sr = File.OpenText(path))
    {
        string s = String.Empty;
        while ((s = sr.ReadLine()) != null)
        {
            yield return new ListKeywords(s);
        }
    }
}

Note that Jon Skeet's C# in Depth offers a great explanation about this in Chapter 6. I imagine he also has some articles or posts on StackOverflow about this topic. As he points out, you want to be careful about modifying this method to pass in a StreamReader (or TextReader as is used in his example) as you would want to take ownership of the reader so it will be properly disposed of. Rather, you would want to pass in a Func<StreamReader> if you have such a need. Another interesting note he adds here - which I will point out because there are some edge cases where the reader will not actually be properly disposed of even if you don't allow the reader to be provided by the caller - it's possible for the caller to abuse the IEnumerable<ListKeywords> by doing something like Keywords().GetEnumerator() - this could result in a memory leak and could even potentially cause security issues if you have security-related code which relies on the using statement to clean up the resource.

IEnumerable has some limitations - see here: http://stackoverflow.com/questions/14143438/want-to-remove-an-item-from-a-ienumerablet-collection to get it to behave as a list you have to call .ToList() — Jordan, Apr 15 '14 at 03:32
You may, however, be better off creating a List CleanList and then adding what you want to keep from the Keywords IEnumerable to the CleanList as you enumerate through the IEnumerable e.g. foreach( var keyword in Keywords ) { if( keyword.IsValid() ) { CleanList.Add( keyword ); } } — Jordan, Apr 15 '14 at 03:36

List store large amounts of data , Not enough memory

1 Answers1