1

I have written C# Job to read from IIS line by line and create the JSON and post the data to DB.The Job is reading 45K lines and inserts operations are very slow. I enabled the parallel programming in job but still the performance was very slow. It took almost 2 Hours to complete the operation.

Here is my code snippet for calling the API:

              Parallel.For(0, logV.Count, counter =>
                    {

                        Console.WriteLine(JsonConvert.SerializeObject(logV[counter]));
                        var URL = WebConfigurationManager.AppSettings["URL"];
                        var client = new RestClient(URL);
                        var request = new RestRequest(Method.POST);
                        request.AddHeader("content-type", "application/json");
                        request.AddHeader("Authorization", "Basic TEST");
                        request.AddParameter("application/json", JsonConvert.SerializeObject(logV[counter]), ParameterType.RequestBody);
                        IRestResponse response = client.Execute(request);
                        Console.WriteLine("Lines read" + counter);
                    });

Here is my API:

   // POST: api/IISLogs
    public async Task<IHttpActionResult> Post(IISLog iislog)
    {
        if (!ModelState.IsValid)
        {
            return BadRequest(ModelState);
        }

        db.IISLogs.Add(iislog);
        await db.SaveChangesAsync();

        return CreatedAtRoute("DefaultApi", new { id = iislog.Id }, iislog);
    }

How can i do multiple inserts in Database using single HTTPS POST Opertaion or how can i convert the method as Batching?

ayush choudhary
  • 51
  • 4
  • So you're making 45K http requests? Do you really need to do the inserts through a web api? – Jasen Oct 27 '18 at 20:29
  • Yes...we don't have DB connection on the server to do bulk inserts – ayush choudhary Oct 27 '18 at 20:52
  • Here's what I would do: Load the log as a single file through the Web API. The API service will save the log to a temporary location then create a "Work Ticket". You can immediately return the response that the log was uploaded and a job had been queued. The Work Ticket will store a job id and the path to the file. You can store this info in a table along with the job state (pending, completed, completed with errors, etc.). – Jasen Oct 27 '18 at 21:24
  • Another program will pull a Work Ticket from the queue and _take as long as it needs_ to insert the log into the database (without the network overhead costs). After the log has been processed it updates the Work Ticket and either terminates or moves onto the next item in the queue. This program can either be a forever-running service or a scheduled task to process a single log. – Jasen Oct 27 '18 at 21:25
  • Do @Jasen comments! Trying to process this in an API request isn't a good idea, putting messages on a queue and then have a console or service application read from the queue is the way to go in these situations. As a rule where I work any request that takes more than 2 seconds to complete (and even that is too long sometimes) should be put on a queue and processed outside of the API – matt_lethargic Nov 01 '18 at 14:29
  • Hi,The idea here is to create splunk dashboard for visualing iis logs real-time.We have connected the Database with splunk for that purpose and using the api to push data to DB.DB calls can't be made directly since web servers does not have connectivity with DB – ayush choudhary Nov 12 '18 at 04:42

0 Answers0