C# DelegatingHandler error

Question

I have the following simple delegating handler that reads a header from the request, checks that the token is in a memory cache, and if not, creates a new response message.

public class SessionHandler : DelegatingHandler
{
    private LogService logService = new LogService();

    public static string AuthToken = "X-Auth-Token";

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        HttpResponseMessage response = null;

        try
        {
            var headers = request.Headers;
            if (headers.Contains(AuthToken))
            {
                string auth = headers.GetValues(AuthToken).First();
                request.Properties[AuthToken] = auth;

                Session session = Cache.GetSession(auth);

                //Session doesn't exist
                if (session == null)
                    response = new HttpResponseMessage(HttpStatusCode.Conflict);
            }

            else
                response = new HttpResponseMessage(HttpStatusCode.Unauthorized);
        }
        catch (Exception ex)
        {
            //
        }

        //Handle the response message
        if (response != null)
        {
            var task = new TaskCompletionSource<HttpResponseMessage>();
            task.SetResult(response);
            return await task.Task;
        }

        return await base.SendAsync(request, cancellationToken);
    }
}

I have logged each and every line and everything is working fine. However my controller is returning a 500 error. And I don't know if the error is in the last return of the handler or somwehere else in the pipeline before reaching the controller.

The controller inherits from this base controller:

public abstract class BaseController : ApiController
{
    protected Session session;

    protected override void Initialize(HttpControllerContext controllerContext)
    {
        try
        {
            logService.Logger().Info("BaseController.Initialize");

            string authToken = controllerContext.Request.Properties[SessionHandler.AuthToken] as string;
            session = Cache.GetSession(authToken);

        }
        catch (Exception ex)
        {
            //
        }

        base.Initialize(controllerContext);
    }

}

The log info ("BaseController.Initialize") is never being reached so the error is occurring somewhere else.

After further debugging, when I comment ma routes as follows:

        config.Routes.MapHttpRoute(
            name: "ApiById",
            routeTemplate: "api/{controller}/{id}",
            defaults: new { id = RouteParameter.Optional },
            constraints: new { id = @"^[0-9]+$" }
            //handler: new SessionHandler()
        );

On executing the following line in the delegating handler:

return await base.SendAsync(request, cancellationToken);

The following exception is thrown:

System.InvalidOperationException: The inner handler has not been assigned.

Why? and how can I resolve this problem?