Best way to pass emails to an rails app

Question

I'm thinking about the architecture of a system which should handle incoming mails and pass it to a rails app which processes the incoming mails. I'm not sure whats the best way to do something like that.

It should work like this:

• Mails are send to something like contact@user_id.myapp.com
• The mailserver accepts the mails and passes them on to the rails app (or stores them and the rails app fetches them)
• The rails app processes the mails (some analyzing then throws them in the db)

It should basically work like a queque. Sudden peaks in mail traffic shouldn't kill the rails app.

I'm not looking for a complete solution. I'm just interested in your opinion. I figured 3 possible options:

• Rails connects via Pop3 to the mailserver and just downloads the messages (probably slow)
• The mailserver pipes the mails with a POST request to the rails APP (probably also too slow, could kill the webserver when encountering a mailbomb)
• The maildir is virtually linked into the rails app server's filesystem (mailserver and app server have to be separate) and rails just reads directly from that.

What do you think is the best approach performance- and security-wise? Am I missing something here? Is there a better way? Do you now some best practice resources?

Thanks!

Answer 1

Here's a post that may help:

Receiving Emails and Attachments with Rails

The article also has links to previous posts on setting up RailsCron.

Answer 2

You can download a free POP/SMTP com component @ www.system-engine.com, which has hooks for integrating into a web application.

Answer 3

I've seen apps do this with POP3 (Spiceworks is one example I can think of). I think it's a decent way to separate the app. server from the mail server, and allows you to let the mail server concentrate on what it does well and freeing the client from the queuing / storage duties for the messages.

re: Security

An issue with POP3 that springs to mind is the default use of cleartext credentials. If you can run it over SSL (mail server dependent), you can mitigate that concern.

re: Perf and scaling

I'm not so sure that POP3 access is going to be all that slow. I'd be wary of doing the integration at the filesystem level, because you may have contention and locking issues (ugh-- think about mounting maildirs over NFS as an example of fun filesystem contention issues) as new mail comes in. POP3 gives you a nice method to atomically access items in the mailbox.

Having multiple consumers running on the same POP3 mailbox at the same time would probably be problematic (if you're trying to scale out to handle more message traffic). For that, you might want to script something on the mail server side to round-robin distribute incoming messages into a group of mailboxes, and tie each consumer to a given mailbox. (You might consider using IMAP for a multiple consumer architecture, but that's just me blowing smoke w/o thinking it thru.)

You're adding more layers and potential bottlenecks, certainly, than just accepting SMTP directly into your code, but you're leveraging all the work that the mail server authors have already done. I'd rather have a scaling problem on a mail server, though, than a scaling problem on custom SMTP server code.

Answer 4

Rails is probably the wrong tool for the job. You would probably want to write a small script that accepted the email on stdin and inserted it into your database. You might be able to use the same ActiveRecord code to do this. Then you just need to set up your MTA to deliver the mail by piping it to your script.

If you need a little more performance, you can rewrite your script to be a daemon that accepts the email via LTMP, which will save the inefficiency of starting a process for every mail.

Answer 5

I'd either simply create a tool that regularly fetches all mails in the defined POP3 accounts, or even better yet (IMHO) fetch it on an event basis.

By event basis I mean something like: You have a website that lists the emails retrieved. Whenever this site is loaded a background task is fired off that fetches unseen mails from the mailbox.

Of course it actually depends on what your system looks like. It's hard to say without actually knowing application.

On the other hand, you'd probably better ask this on http://stackoverflow.com

Answer 6

Instead of POP (if you go that route), use IMAP(S). That way you can leave things on the server "read" once they've been acted on.