ASP.NET WebApi always returns 401 Unauthorized

Question

I am trying to create a new endpoint in my WebApi which will be called from another Server. So, I want to set up the S2S JWT bearer based authentication for my WebApp.

The startup.cs code looks like this

        public void Configuration(IAppBuilder app)
    {
        InitializeLoggers();
        HttpConfiguration httpConfiguration = new HttpConfiguration();
        WebApiConfig.Register(httpConfiguration);
        app.Use<JwtS2SAuthMiddleware>();
        app.UseWebApi(httpConfiguration);
    }

I have added the new line here app.Use<JwtS2SAuthMiddleware>();

Inside the JwtS2SAuthMiddleware, I do the following checks

1. context.Request.Path.Value.StartsWith("/newendpoint");
2. Get the Authorization bearer token from request header
3. Validate the audience, issuer, issuer signing keys using the JwtSecurityTokenHandler.ValidateToken()
4. Verify the appId contained in JWT is in the allowed list of App Id's

Upon debugging using a request from Fiddler, I see all the above checks pass but still the response is 401 Unauthorized with a message

{"message":"Authorization has been denied for this request."}

Can someone help if I would need to anything else after doing the checks in my JwtS2SAuthMiddleware class ? Thanks.