0

I am trying to read a very large csv file from s3 bucket of row size 100000 each. I am using yield return to return the object when the specific row count is achieved. Then, I'm processing the data and fetching again, then it throws the exception. I did some debugging and saw that if row is not fetched for more than 20 seconds, it is throwing the exception. How, to keep the reader object for longer period of time?

using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture)
{
 while(csv.Read())
{
 //data insertion to table logic
if(table.Rows.Count == 100000)
{
yield return table;
}
}
}

I am calling this method inside a foreach loop. Also, let me know if there is a way to implement multipart file download from Amazon S3 bucket.

I tried converting the using block to using statement and passing leaveOpen param as true for CsvReader object.

Indranil Saha
  • 1
  • what exception does it throw? object reference not set to instance of object? are you sure that this is the problem? did you look if the reader gets disposed once you yield a value out, because you left the using block? – Michael Schönbauer Aug 10 '23 at 11:55
  • The connection needs to be modified. Where is your connection? – jdweng Aug 10 '23 at 12:39

1 Answers1

0

Increase Timeout or Chunk Size: If the reader object is timing out after 20 seconds, you might want to consider increasing the timeout period, or alternatively, fetching a larger chunk of rows before processing. This way, you reduce the overhead of fetching new data frequently.

Joseph Freeman
  • 7
  • 2