I'm new to Azure WebJobs and I have a basic question. I have a console app deployed as a WebJob, by nature console apps use static a lot, and I have some local static variables as shown below, would I run into trouble with multiple threads updating these variables simultaneously by the WebJob?
class Program
{
private static MyService _service;
private static int _counter;
static void Main()
{
...
}
private static void Method1() { .... }
}
Since you tagged the question with azure-webjobs-sdk and I remember that you had another question a few days ago about the SDK, I will assume your webjob uses the SDK and the multithreading is caused by the fact that we running triggered functions in parallel.
The first part of my answer is not necessarily web jobs related:
class Program
{
private static MyService _service;
private static int _counter;
private static readonly object _lock = new object();
// The 3 blocks of code are thread safe by themselves
// but put together, there is no guarantee. Avoid
// multiple locks if you can
private static void Method1()
{
// 1. You can use a lock
lock(_lock)
{
// All the code here will be executed by a single thread.
// If another thread tries to get the lock, it will have
// to wait until the lock is released
_service.DoSomething();
}
// 2. For atomic operations you can use Interlocked
Interlocked.Increment(ref _counter);
// 3. For conditional locking
if (_service.SomeBooleanProperty)
{
lock(_lock)
{
// Check again to see the condition still holds
// because it might have changed between the
// execution of the first if and the lock
if (_service.SomeBooleanProperty)
{
// Execute the non thread safe code
}
}
}
}
}
Reference to the Interlocked class
Now I'm talking WebJobs SDK:
If you want to control the concurrency for processing, you can set the Queue.BatchSize property. By default is 16. If you set it to 1, everything runs sequential. See the reference here. If you are having concurrency issues with multi instance concurrency (multiple instances of the same job) then blob leases are the way to go.
No, not unless you specifically write multi-threaded code in your console application. When the WebJob is invoked in Azure, it will get it's own process. For example, here is a screen capture from process explorer for a WebJob called ConsoleTestApp.exe. Notice that it's PID is separate from the W3WP process your Website runs in.
If you specifically want to limit the batch size to one (within the Azure WebJob context) then you can do the following;
static void Main()
{
InitializeStorage();
JobHostConfiguration config = new JobHostConfiguration();
config.Queues.BatchSize = 1; // the Azure default is 16
var host = new JobHost(config);
host.RunAndBlock();
}
Read the Parallel execution with Queues bit in this article for more info.
What do you mean by the "threads"?
Some actual threads that are spawned by your code? (You do not show any in your sample code.) Then indeed, you get into trouble when you access the static variables. Just as with any other multi-threaded code.
But I assume, you actually do not mean threads, but separate running instances of the job. These are separate processes, so each have their own copy of the static variables. They cannot even access each other memory.
Note that the WebJob is just an application, as any other. There's no magic. If you run the same application multiple times, each instance have its own memory allocated for the static variables.
Multiple instances of the job will run in different threads, in the same app domain. This means that static variables are not thread safe.
This is easy to test. It's also the reason why if you register a TextWriterTraceListener with System.Diagnostics.Trace.Listeners and then write to it with Trace.WriteLine from multiple jobs, and finally remove it, you will get cross talk.
by nature console apps use static a lot
Console apps aren't inherently forced into using static variables, and neither are WebJobs. Create an instance of a class to do some processing and call methods of that class. Here is a very basic example of this:
class Program
{
public int Start(string[] args)
{
// You can use non-static variables here
Console.WriteLine("There were {0} arguments.", args.Length);
return 0;
}
static int Main(string[] args)
{
return new Program().Start(args);
}
}
We need to see the processing of WebJobs by SDK similar to ASP.Net request processing model. Like in ASP.Net where eash request is processed in different thread, here in WebJobs SDK each and every invocation of job happens in different thread. If you have static variables which are manipulated by web jobs, there are chances of problems. If there is only one WebJob in your application you can avoid the problems by setting batchsize to 1 so that only one job will be running at a time and it can manipulate static variables.
But if there are multiple web job functions and all those uses same static variables the batch size wont help as at least one invocation from all the types will be running at a time and all those manipulate same static variable.
We have same problem with legacy code base which uses static variables and has memory leaks. We are planning to spawn another exe to do actual processing from the webjobs exe. The child exe just accepts the parameters from the webjob exe which are needed for the job and it will close itself after completion of single job. So that the memory leaks and static variables in legacy code wont cause any trouble. The webjobs exe can simply listen to the console.writes from the child exe and write into the same stream so that is visible in the webjobs portal.
