Queue with multiple columns?

Question

I am looking for a library or a service that lets me push data, and each time there will be several other columns. Right now, I am currently using IronMQ to push json to the queue and then parsing the json and grabbing the attributes. However, I am wondering if there's a queue library or service that lets me just push all the attributes in separate columns so that later on, I can do a SQL query in the queue.

I've thought about Python and Sqlite3 to do this (what difference would it be compared to Amazon SQS or IronMQ) which would make this very easy.

Sounds like you want a database, not a queue? – Travis Reeder Oct 21 '14 at 22:53 — Travis Reeder, Oct 21 '14 at 22:53

Paul Sasik · Accepted Answer · 2014-10-22T15:48:27.007

The difference between message queuing technologies (e.g. IronMQ, RabbitMQ, Amazon SQS etc.) and RDBMSs (e.g. Sqlite, Sql Server, PostGres etc.) is that if you use RDBMS tables as queues and want to allow querying then you will have to manage the LIFO queuing process yourself. Queues (with some exceptions) enforce a LIFO process while Database tables are random access and do not enforce how or when data is inserted, modified or removed. Those rules are handled in the business layer. (You mentioned using Python and Sqlite. Sqlite would be your queue/data store and a Python app would perform the logic of managing the LIFO queue as well as random querying.)

RDBMSs are often used as intermediary message stores and from your question I'm guessing that a LIFO queue isn't even that important since you seem to be interested in random access above everything else. It's hard to tell without knowing more about what you're designing. But if you want something that works like a Queue and a Database off-the-shelf then check out Oracle AQ.

From the linked page:

Because Oracle Streams Advanced Queuing is implemented in database tables, all operational benefits of high availability, scalability, and reliability are also applicable to queue data. Standard database features such as recovery, restart, and security are supported by Oracle Streams Advanced Queuing. You can use database development and management tools such as Oracle Enterprise Manager to monitor queues. Like other database tables, queue tables can be imported and exported.

Messages can be queried using standard SQL. This means that you can use SQL to access the message properties, the message history, and the payload. With SQL access you can also do auditing and tracking. All available SQL technology, such as indexes, can be used to optimize access to messages.

Queue with multiple columns?

1 Answers1