Identity Manager throws 401 unauthorized

Question

I am using following configuration for my Identity Manager, everytime I pass my access token - it throws 401 (Unauthorized) app.Map("/admin", adminApp => {

        var factory = new IdentityManagerServiceFactory();
        factory.ConfigureSimpleIdentityManagerService("AspId");

        var securityConfig = new ExternalBearerTokenConfiguration
        {
            Issuer = "https://localhost:44333/core",
            Audience = "https://localhost:44333/core/resources",
            Scope = "idmgr",
            SigningCert = Certificate.Get(),
        };

        adminApp.UseIdentityManager(new IdentityManagerOptions()
        {
            Factory = factory,
            SecurityConfiguration = securityConfig
        });
    });

My acesss token which I pass as AUthorization Bearer looks like

{
"iss": "https://localhost:44333/core",
"aud": "https://localhost:44333/core/resources",
"exp": 1449995870,
"nbf": 1449995510,
"client_id": "roclient",
"scope": [
"email",
"idmgr",
"openid",
"phone",
"profile"
],
"sub": "58067d52-64bf-4447-a928-d3a1316bd677",
"auth_time": 1449995509,
"idp": "idsrv",
"role": [
"admin",
"IdentityManagerAdministrator"
],
"amr": [
"password"
]
}

Is it something I am not doing right ?