in my current ASP.net MVC 3.0 project i am stuck with a situation.
- I have four .txt files each has approximatly 100k rows of records
- These files will be replaced with new files on weekly bases.
I need to Query data from these four text files, I am not able to choose the best and efficient way to do this.
3 ways I could think
- Convert these text files to XML on a weekly basis and query it with Linq-XML
- Run a batch import weekly from txt to SQL Server and query using Linq-Entities
- avoid all conversions and query directly from text files.
Can any one suggest a best way to deal with this situation.
update:
Url of the Text File

I should connect to this file with credentials.
once i connect successfully, I will have the text file as below with Pipeline as Deliminator
This is the text file

Note: First two lines of the text file are headers of the File.