Getting 401 not_authorised when sending a request to a RabbitMQ Exchange

Question

I'm getting the following response when trying to publish to a RabbitMQ exchange whose virtual host the user has access to:

{
    "error": "not_authorised",
    "reason": "Not management user"
}

I can give the user the management tag but that would let him access the UI which is not what we want.