1

I am working on an asp.net mvc 5 web application , deployed inside IIS-8, and i have a method inside my application to perform a long running task which mainly scans our network for servers & VMs and update our database with the scan results. method execution might last between 30-40 minutes to complete on production environment. and i am using a schedule tool named Hangfire which will call this method 2 times a day.

here is the job definition inside the startup.cs file, which will call the method at 8:01 am & 8:01 pm:-

public void Configuration(IAppBuilder app)
{
    var options = new SqlServerStorageOptions
    {
        PrepareSchemaIfNecessary = false
    };
    GlobalConfiguration.Configuration.UseSqlServerStorage("scanservice",options);

    RecurringJob.AddOrUpdate(() => ss.Scan(), "01 8,20 ***");
}

and here is the method which is being called twice a day by the schedule tool:-

public void Scan()
{
    Service ss = new Service();
    ss.NetworkScan().Wait();
}

Finally the method which do the real scan is (i only provide a high level description of what the method will do):-

public async Task<ScanResult> NetworkScan()
{
    // retrieve the server info from the DB
    // loop over all servers & then execute some power shell commands to scan the network & retrieve the info for each server one by one...
    // after the shell command completed for each server, i will update the related server info inside the DB

currently i did some tests on our test environment and every thing worked well ,, where the scan took around 25 seconds to scan 2 test servers.but now we are planning to move the application to production and we have around 120++ servers to scan. so i estimate the method execution to take around 30 -40 minutes to complete on the production environment. so my question is how i can make sure that this execution will never expire , and the ScanNetwork() method will complete till the end?

Dangerous
  • 4,818
  • 3
  • 33
  • 48
John John
  • 1
  • 72
  • 238
  • 501
  • 2
    Shouldn't it run until it is done or runs across an error causing it to return early? – Dylan Oct 02 '15 at 15:30
  • @Dylan can you adivce more what do u exactly mean ? i want to make sure that the operation will never expires if it runs for 30-40 minutes ? – John John Oct 02 '15 at 15:32
  • 2
    Why are you worried that the method will time out? It should run until the method is done , or it runs into an issue and ends. I don't believe a method has a time limit. – Dylan Oct 02 '15 at 15:33
  • @Dylan but i was assuming that IIS thread will timeout after certain time – John John Oct 02 '15 at 20:46
  • 3
    The default timeout of the App pool is set to 20 min. You can change it from the advanced settings on the app pool your using. A windows service may be better for this so you dont have to worry about it timing out. Heck you could make it in a console app, and run it from Task Scheduler too. – Dylan Oct 02 '15 at 20:55
  • 2
    Dylan is right. If you run something like this in the Web Server context - don't do it. If the website has to start something like this, create endpoint application and start it from web app. Looks like your website doesn't even monitor the feedback. If your website monitors output of your scan, just write updates somewhere and make website grab it periodically. Or use SignalR for back-feed – T.S. Oct 02 '15 at 21:50
  • 2
    Increase also the `SqlServerOptions.InvisibilityTimeout` (defaults to 30 minutes), or just update to Hangfire 1.5.0 (I removed that timeout) to prevent other worker from fetching the same background job after a timeout. – odinserj Oct 03 '15 at 07:09
  • Consider also to use `IJobCancellationToken` parameter (or simply `CancellationToken` and its `ThrowIfCancellationRequested` method, if you update to 1.5.0) and pass it to your `NetworkScan` method to be able to cancel scanning on app shutdown and prevent shutdown timeouts. – odinserj Oct 03 '15 at 07:16
  • @odinserj i am using hangfire 1.4.6 and i think it will handle managing the shutdown ,, but my question is how i can avoid any timeout during the method execution...di u get my point thnaks. – John John Oct 03 '15 at 13:16
  • @T.S. i know that doing this inside a web application might not be the 100% robust solution,, but my question was is how i can improve this ? and what are the problems that i will face if i keep doing this long running process from a web application.. – John John Oct 03 '15 at 13:19
  • 1
    This is the answer - the problem is - you taking resources from your web app, leaving it less resources and potentially making it slower. The timeout thing you're talking about will not happen, if lets say, you start a process in web context on a separate thread - it will run all the way to the end until thread stops. – T.S. Oct 03 '15 at 16:24
  • @T.S. thanks for the reply,, so you mean the web context process will run it the end even if the app pool timeout is set to 20 minute ? i did not get your point ? can you adivce more on this please ? – John John Oct 03 '15 at 23:41
  • 1
    You probably talking about idle timeout. But your thread is not idle. This timeout only removes idle workers. Your worker will be active. – T.S. Oct 04 '15 at 03:00
  • @T.S. but i always read that execution a long running process inside IIS might cause timeouts,, so not sure why in my case i will not receive any timeouts ? – John John Oct 05 '15 at 11:15
  • Again, if you have a request and before getting response out, you run a long job - you may cause timeout. But if you send back response immediately and run your long job on background thread - in this case your thread will finish without timeout. – T.S. Oct 05 '15 at 13:54
  • @T.S. thanks for the reply but can you advice please what do you mean by "But if you send back response immediately.,,,," waht do you mean by send back response immediately ? – John John Oct 06 '15 at 00:20
  • You're on the web server. The nature of it is that it receives request and sends response. If your request requires a long running job, you may receive `web request timed out`. So, in these cases you start processing long running job on the background thread and then send response back. This will make response going out to the client while LRJ process as well – T.S. Oct 06 '15 at 01:12
  • Possible duplicate of [How to prevent a Hangfire recurring job from restarting after 30 minutes of continuous execution](https://stackoverflow.com/questions/33427069/how-to-prevent-a-hangfire-recurring-job-from-restarting-after-30-minutes-of-cont) – Owen Pauling Mar 26 '19 at 09:52

1 Answers1

3

Instead of worrying about your task timing out, perhaps you could start a new task for each server. In this way each task will be very short lived, and any exceptions caused by scanning a single server will not effect all the others. Additionally, if your application is restarted in IIS any scans which were not yet completed will be resumed. With all scans happening in one sequential task this is not possible. You will likely also see the total time to complete a scan of your entire network plummet, as the majority of time would likely be spent waiting on remote servers. 

public void Scan()
{
    Service ss = new Service();
    foreach (var server in ss.GetServers())
    {
        BackgroundJob.Enqueue<Service>(s => s.ServerScan(server));
    }
}

Now your scheduled task will simply enqueue one new task for each server.

Austin Morton
  • 1,150
  • 1
  • 8
  • 6