asp.net mvc 4 skip middleware if there is no [Authorize] attribute present

Question

I have a middleware that should only be applied in the presence of [Authorize] attribute. I need to be able to do something like

if( !authorizeIsPresent ) {
   Next.Invoke(context); // Skip this middleware
}
else {
   Do normal stuff when authorizeIsPresent
}

I've tried to look at the context property, but can't seem to find anything. Can someone point me in the right direction?