There is a cloud solution based on the popular RabbitMQ message broker:
CloudAMQP - RabbitMQ as a Service at http://www.cloudamqp.com/
It is available in multiple clouds (including AWS). RabbitMQ is a multi-protocol broker, and all CloudAMQP plans support the AMQP, AMQPS, HTTPS, STOMP and MQTT protocols.
AMQP, STOMP and MQTT use message push, with permanent or periodic connections to the server.
AMQP/STOMP/MQTT client libraries should be available for all major mobile platforms (see http://www.rabbitmq.com/devtools.html). For a comparision of protocols: Choosing Your Messaging Protocol: AMQP, MQTT, or STOMP.
The entry level plan with dedicated server (Big Bunny) supports around one million queued messages (if the devices are not connected, pending notifications wait on the server).
About queue resource usage:
In RabbitMQ, queues are quite cheap. They're effectively lightweight
Erlang processes, and you can run tens to hundreds of thousands of
queues on a single commodity machine
(from https://stackoverflow.com/a/7040605/80901)
(I am a developer of STOMP message broker client libraries Delphi / Free Pascal for Windows and Linux)