Ninject scope when running Azure Webjob in parallel

Question

I have a console application as my webjob to process notifications inside my application. The processes are triggered using a queue. The application interacts with a SQL Azure database using entity framework 6. The Process() method that's being called reads/write data to the database.

I'm getting several errors when the queue messages are processed. They never get to the poison queue since they are reprocessed successfully after 2-3 times. Mainly the errors are the following:

An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll

Error: System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown.

The default batch size is 16, so the messages are processed in parallel.

My guess is that the Ninject setup for processing messages in parallel is wrong. Therefore, when they are processed at the same time, there are some errors and eventually they are processed successfully.

My question is: Does this setup look ok? Should I use InThreadScope() maybe since I don't know the parallel processing is also multi-threaded.

Here's the code for my application.

Program.cs

namespace NotificationsProcessor
{
    public class Program
    {
        private static StandardKernel _kernel;
        private static void Main(string[] args)
        {
            var module = new CustomModule();
            var kernel = new StandardKernel(module);
            _kernel = kernel;

            var config =
                new JobHostConfiguration(AzureStorageAccount.ConnectionString)
                {
                    NameResolver = new QueueNameResolver()
                };
            var host = new JobHost(config);
            //config.Queues.BatchSize = 1; //Process messages in parallel
            host.RunAndBlock();
        }

        public static void ProcessNotification([QueueTrigger("%notificationsQueueKey%")] string item)
        {
            var n = _kernel.Get&ltINotifications>();
            n.Process(item);
        }

        public static void ProcessPoison([QueueTrigger("%notificationsQueueKeyPoison%")] string item)
        {
            //Process poison message. 
        }
    }
}

Here's the code for Ninject's CustomModule

namespace NotificationsProcessor.NinjectFiles
{
    public class CustomModule : NinjectModule
    {
        public override void Load()
        {
            Bind&ltIDbContext>().To&ltDataContext>(); //EF datacontext
            Bind&ltINotifications>().To&ltNotificationsService>();
            Bind&ltIEmails>().To&ltEmailsService>();
            Bind&ltISms>().ToSmsService>();
        }
    }
}

Code for process method.

    public void ProcessMessage(string message)
    {
        try
        {
            var notificationQueueMessage = JsonConvert.DeserializeObject&ltNotificationQueueMessage>(message);
            //Grab message and check if it has to be processed
            var notification = _context.Set().Find(notificationQueueMessage.NotificationId);

            if (notification != null)
            {
                if (notification.NotificationType == NotificationType.AppointmentReminder.ToString())
                {
                    notificationSuccess = SendAppointmentReminderEmail(notification); //Code that sends email using the SendGrid Api
                }
            }
        }
        catch (Exception ex)
        {
            _logger.LogError(ex + Environment.NewLine + message, LogSources.EmailsService);
            throw;
        }                   
    }

Update - Added Exception

The exception is being thrown at the Json serializer. Here's the stack trace:

Error: System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown. at System.String.CtorCharCount(Char c, Int32 count) at Newtonsoft.Json.JsonTextWriter.WriteIndent() at Newtonsoft.Json.JsonWriter.AutoCompleteClose(JsonContainerType type) at Newtonsoft.Json.JsonWriter.WriteEndObject() at Newtonsoft.Json.JsonWriter.WriteEnd(JsonContainerType type) at Newtonsoft.Json.JsonWriter.WriteEnd() at Newtonsoft.Json.JsonWriter.AutoCompleteAll() at Newtonsoft.Json.JsonTextWriter.Close() at Newtonsoft.Json.JsonWriter.System.IDisposable.Dispose() at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer) at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, Formatting formatting, JsonSerializerSettings settings) at Core.Services.Communications.EmailsService.SendAppointmentReminderEmail(Notificaciones email) in c:\ProjectsGreenLight\EAS\EAS\EAS\Core\Services\Communications\EmailsService.cs:line 489 at Core.Services.Communications.EmailsService.ProcessMessage(String message) in c:\ProjectsGreenLight\EAS\EAS\EAS\Core\Services\Communications\EmailsService.cs:line 124 at Core.Services.NotificacionesService.Process(String message) in c:\ProjectsGreenLight\EAS\EAS\EAS\Core\Services\NotificacionesService.cs:line 56

Answer 1

Since you're receiving OutOfMemoryExceptions and StackOverflowExceptions, i suggest that there may be a recursive or deeply nested method. It would be extremely helpful if you'd have a stacktrace to the exception, sadly that's not the case for StackOverflowExceptions. However, OutOfMemoryException has a stack trace, so you need to log this and see what it says / add it to your question. Also, as far as the StackoverflowException goes, you can also try this.

Scoping

You should not use .InThreadScope() in such a scenario. Why? usually the thread-pool is used. Thread's are reused. That means, that scoped objects live longer than for the processing of a single messages. Currently you are using .InTransientScope() (that's the default if you don't specify anything else). Since you are using the IDbContext in only one place, that's ok. If you'd wanted to share the same instance across multiple objects, then you'd have to use a scope or pass along the instance manually.

Now what may be problematic in your case is that you may be creating a lot of new IDbContexts but not disposing of them after you've used them. Depending on other factors this may result in the garbage collector taking longer to clean-up memory. See Do i have to call dispose on DbContext.

However i grant that this won't fix your issue. It might just speed up your application. Here's how you can do it anyway:

What you should do is: Have INotifications derive from IDisposable:

public interface INotifications : IDisposable 
{
    (...)
}

internal class Notifications : INotifications
{
    private readonly IDbContext _context;

    public Notifications(IDbContext context)
    {
        _context = context;
    }

    (...)

    public void Dispose()
    {
        _context.Dispose();
    }
}

and alter your calling code to dispose of INotifications:

    public static void ProcessNotification([QueueTrigger("%notificationsQueueKey%")] string item)
    {
        using(var n = _kernel.Get<INotifications>())
        {
           n.Process(item);
        }
    }