8

What is the best way to load huge text file data in delphi? Is there any component that can load text file superfast?

Let's say I have a text file contains database and stored in fix length format. It contains 150 field with each at least 50 characters. 1. I need to load it into memory 2. I need to parse it and probably store it in a memdataset for processing

My questions: 1. Is it enough if I use TStringList.loadFromFile method? 2. Is there any other better component to manipulate the text file? 3. Should I use low level reading from textfile?

Thank you in advance.

WishKnew
  • 649
  • 2
  • 10
  • 22

4 Answers4

12

TStringList is never the optimal way of working with lots of text, but it's the simplest. If you've got small files on your hands you can use TStringList without issues. Even if you have large files (not huge files) you might implement a version of you algorithm using TStringList for testing purposes, because it's simple and easy to understand.

If your files are large, as they probably are since you call them "databases", you need to look into alternative technologies that will enable you to read only as much as you need from the database. Look into:

  • TFileStream
  • Memory mapped files.

Don't look at the old "file" based API's still available in Delphi, they're plain old.

I'm not going to go into details on how to access text using those methods because we've recently had two similar questions on SO:

How Can I Efficiently Read The FIrst Few Lines of Many Files in Delphi

and

Fast Search to see if a String Exists in Large Files with Delphi

Community
  • 1
  • 1
Cosmin Prund
  • 25,498
  • 2
  • 60
  • 104
2

Since you have a fixed length that you're working with, you can build an access class based on TList with a TWriter and TReader that will take your records into account. You'll have none of the overhead of a TStringList (not that it's a bad thing, but if you don't need it, why have it) and you can build in your own access to records into the class. Ultimately it depends on what you are trying to accomplish with the data once you have it loaded into memory. While TStringlist is easy to use, it isn't as efficient as "rolling your own".

However, efficiency in data manipulation may not be that much of an issue, as you are using text files to hold a database. If you just need to read in and make decisions based on data in the file, the more flexible TList may be overkill.

Doug Johnson-Cookloose
  • 530
  • 4
  • 17
  • I doubt the length of text on each line is fixed; The OP says `It contains 150 field with each at least 50 characters`. I think the fix quantity is the number of fields. – Cosmin Prund Feb 27 '11 at 17:59
1

I recommend to adhere to TStringList if you find it convenient for your problem. Optimization is another thing that should be done later.

As for TStringList the optimization is to declare a descendant class that overrides TStrings.LoadFromStream method - you can make it practically as fast as possible, taking into account the structure of your files.

kludg
  • 27,213
  • 5
  • 67
  • 118
1

It is not entirely clear from your question why you need to load the entire file into memory, prior to then going on to create an in-memory data set.... are you conflating the two issues? (i.e. because you need to create an in-memory data set you think you first need to load the source data entirely into memory? Or is there some initial pre-processing of the source file which is only possible with the entire file loaded in memory (this is unlikely and even if this is the case, it isn't necessary with a navigable stream object such as a TFileStream).

But I think the answer you are looking for is right there in the question....

If you are loading this file in order to parse it and populate/initialise a further data structure (the data set) for further processing, then using an existing high level data structure is an unnecessary and potentially costly (in terms of time) step.

Use the lowest level means of access that provides the capabilities you need.

In this case a TFileStream will likely provide the best balance of convenience and ease of use.

Deltics
  • 22,162
  • 2
  • 42
  • 70
  • I need to compare from to files. I have two input file, one as master database, and another is file to be compared. I need to list all the fields in the master database which match with certain values in the file to be compared. – WishKnew Feb 28 '11 at 10:18
  • This explains what you are doing, but it still doesn't explain why you have to do it in the way that you appear to have decided. It still isn't clear to me that you cannot use a stream to do what you have described. If performance is a concern, then I'd implement first using a TFileStream - if performance does turn out to be an issue you could convert to a TMemoryStream almost instantly. If necessary *then* you can look at more sophisticated structures based on where your specific performance problems actually come from. – Deltics Mar 02 '11 at 00:46