Problems forwarding Mailgun's incoming email to own SMTP server using HTTP POST route

Question

Recently I had to hide my IP address using CloudFlare, but with my MX record pointed to my own domain, I can't.

I know that Mailgun can forward all the emails to user@example.com to Gmail, but I don't want that. I hope the email can be forwarded example.com's own SMTP server.

I decided to use Mailgun catch_all() and forward("https://exmaple.com/incoming-email?key=SECRET_KEY"), and wrote some node.js code for that.

var nodemailer = require('nodemailer');
var express = require('express');
var multer = require('multer');
var router = express.Router();

var KEY = 'SECRET_KEY';
var ADDITIONAL_HEADERS = ['timestamp', 'token', 'signature', 'mime-version'];
var transporter = nodemailer.createTransport('smtp://127.0.0.1');

var storage = multer.memoryStorage();
var upload = multer({
  storage: storage
});

router.post('/', upload.any(), function (req, res, next) {
  if (!req.query.key || req.query.key !== KEY) {
    res.sendStatus(403);
    return;
  }
  console.log('Incoming mail: ', req.body.From);
  req.body.to = req.body.To;
  req.body.cc = req.body.Cc;
  req.body.from = req.body.From;
  req.body.date = req.body.Date;
  req.body.references = req.body.References;
  req.body.replyTo = req.body['Reply-To'];
  req.body.inReplyTo = req.body['In-Reply-To'];
  req.body.messageId = req.body['Message-Id'];
  req.body.html = req.body['body-html'];
  req.body.text = req.body['body-plain'];
  req.body.headers = {};
  for (var key in req.body) {
    if (key.toLowerCase().indexOf('X-') !== -1 ||
      ADDITIONAL_HEADERS.indexOf(key.toLowerCase()) !== -1) {
      req.body.headers[key] = req.body[key];
    }
  }
  if (req.files) {
    req.body.attachments = req.files;
    req.body.attachments.forEach(function (attachment) {
      attachment.filename = attachment.originalname;
      attachment.contentType = attachment.mimetype;
      attachment.content = attachment.buffer;
    });
  }
  transporter.sendMail(req.body, function (err, info) {
    if (err) {
      console.log(err);
      res.status(400).send(err);
      return;
    }
    console.log(info);
    res.send(info);
  });
});

module.exports = router;

As you can see, the code is able to forward any incoming emails to localhost SMTP server. Using the code about, I can receive emails using user@example.com. The email received will have correct from, to, cc, subject and body and even attachments.

However, a day later I am forced to be unsubscribed from Arch Linux mailing lists, and a person emailed to me that:

It appears that your mail server is misconfigured and sends at least some mails it receives from our mailing lists back to the list. It also removes some headers in the process which help mailman to detect such behaviour. Since those headers are missing, mailman will send the mail out once again to everyone, leading to a never ending loop.

After I called out for help, I have also received some emails indicate that I was unable to receive any emails from mailing lists in the period.

Why could it go wrong? What are the "some headers" in the email removed in the forwarding / wrapping process? Is it Mailgun's problem or my own problem? If it's my problem, what could be added to the code above? Thanks.

Try a working example: https://www.noodl.io/market/product/P201601221424994/email-composer-send-emails-with-nodejs-and-mailgun — WJA, Jan 25 '16 at 00:56

score 1 · Accepted Answer · answered Jan 17 '16 at 08:50

I found that when the email is Bcc to you, then the mail will be looked like:

To: announce@example-mailing-list.com
Bcc: you@example.com

So that when using the code in the question, the mail will be handle by the node.js code, then the mail will be sent back to announce@example-mailing-list.com using smtp://127.0.0.1, causing infinite loop.

I have updated my code to find all the email addresses of my domain from the recipient list and set them as To field, and the infinite loop no longer presents.

For example, the recipient list will be looked like: announce@example-mailing-list.com, you@example.com, so I am finding all email addresses ending with @example.com and make them To field.

Caveat: some information will be missing though, like I can no longer see the original To field of an email. However, you can append the original To field to any fields, like subject.

var nodemailer = require('nodemailer');
var express = require('express');
var multer = require('multer');
var router = express.Router();

var MY_DOMAIN = 'example.com';
var KEY = 'SECRET_KEY;
var ADDITIONAL_HEADERS = ['timestamp', 'token', 'signature', 'mime-version',
  'return-path', 'dkim-signature'
];
var transporter = nodemailer.createTransport('smtp://127.0.0.1');

var storage = multer.memoryStorage();
var upload = multer({
  storage: storage
});

router.post('/', upload.any(), function (req, res, next) {
  if (!req.query.key || req.query.key !== KEY) {
    res.sendStatus(403);
    return;
  }
  console.log('Incoming mail: ', req.body.From);
  req.body.to = req.body.recipient.split(', ').filter(email => email.indexOf('@' + MY_DOMAIN) !== -1);
  req.body.from = req.body.From;
  req.body.date = req.body.Date;
  req.body.references = req.body.References;
  req.body.replyTo = req.body['Reply-To'];
  req.body.inReplyTo = req.body['In-Reply-To'];
  req.body.messageId = req.body['Message-Id'];
  req.body.html = req.body['body-html'];
  req.body.text = req.body['body-plain'];
  req.body.headers = {};
  for (var key in req.body) {
    if (key.toLowerCase().indexOf('x-') !== -1 ||
      ADDITIONAL_HEADERS.indexOf(key.toLowerCase()) !== -1) {
      req.body.headers[key] = req.body[key];
    }
  }
  console.log('Params: ' + JSON.stringify(req.body));
  if (req.files) {
    req.body.attachments = req.files;
    req.body.attachments.forEach(function (attachment) {
      attachment.filename = attachment.originalname;
      attachment.contentType = attachment.mimetype;
      attachment.content = attachment.buffer;
    });
  }
  transporter.sendMail(req.body, function (err, info) {
    if (err) {
      console.log(err);
      res.status(400).send(err);
      return;
    }
    console.log(info);
    res.send(info);
  });
});

module.exports = router;

score 0 · Answer 2 · answered Jan 14 '16 at 19:50

An email does not only have a To header, but it also has the SMTP RCPT TO value. This value may be different from the To header that is contained inside the email.

See: In SMTP, must the RCPT TO: and TO: match?

Your code seems to use the mail content and does not appear to handle the transport recipient (SMTP RCPT TO) separately. I do not know if mailgun provides this value.

Just submit the mail to your smtp server and some library along the way probably extracts the recipient from the To header, however in the case of mailing lists this is often incorrect.

Thanks! I will give a look Mailgun POST form again when I am home. — Saren Arterius, Jan 15 '16 at 00:34

Problems forwarding Mailgun's incoming email to own SMTP server using HTTP POST route

2 Answers2