0

in my current ASP.net MVC 3.0 project i am stuck with a situation.

  1. I have four .txt files each has approximatly 100k rows of records
  2. These files will be replaced with new files on weekly bases.

I need to Query data from these four text files, I am not able to choose the best and efficient way to do this.

3 ways I could think

  1. Convert these text files to XML on a weekly basis and query it with Linq-XML
  2. Run a batch import weekly from txt to SQL Server and query using Linq-Entities
  3. avoid all conversions and query directly from text files.

Can any one suggest a best way to deal with this situation.

update:


Url of the Text File URL to access the txt File with credentials
I should connect to this file with credentials.

once i connect successfully, I will have the text file as below with Pipeline as Deliminator
This is the text file


Data in Text File is as this Now i have to look up for the field highlighted in yellow and get the data in that row.

Note: First two lines of the text file are headers of the File.

HaBo
  • 13,999
  • 36
  • 114
  • 206
  • Are there delimiters you can use in the records to break them up accordingly and just convert the data to objects? – ryoung Feb 28 '12 at 15:06
  • spaces are the delimiters- I have done this before with another txt to SQL Server as spaces were not really helping me i went with character count. – HaBo Feb 28 '12 at 15:23
  • @ryoung i found that pipeline(|) is used as deliminator – HaBo Mar 07 '12 at 14:42

1 Answers1

0

Well As i Found a way my self. Hope this will be useful for any who are interested to get this done.

string url = "https://myurl.com/folder/file.txt";
WebClient request = new WebClient();
                request.Credentials = new NetworkCredential(ConfigurationManager.AppSettings["UserName"], ConfigurationManager.AppSettings["Password"]);
                Stream s = request.OpenRead(url);
                using (StreamReader strReader = new StreamReader(s))
                {
                    for (int i = 0; i <= 1; i++)
                        strReader.ReadLine();
                    while (!strReader.EndOfStream)
                    {
                        var CurrentLine = strReader.ReadLine();
                        var count = CurrentLine.Split('|').Count();
                        if (count > 3 && CurrentLine.Split('|')[3].Equals("SearchString"))
                        {
                            #region Bind Data to Model
                            //var Line = CurrentLine.Split('|');
                            //CID.RecordType = Line[0];
                            //CID.ChangeIdentifier = Line[1];
                            //CID.CoverageID = Convert.ToInt32(Line[2]);
                            //CID.NationalDrugCode = Line[3];
                            //CID.DrugQualifier = Convert.ToInt32(Line[4]); 
                            #endregion
                            break;
                        }
                    }
                    s.Close();
                }
                request.Dispose();
HaBo
  • 13,999
  • 36
  • 114
  • 206