1

I am a little desperate with a problem in our web server. Our web is a community with a lot of users and activity. We send to users personalized mails of the activity of their interest. We have a problem in the script that handles these mailings.

Our Bulk email script fails when it has thousands of emails to send. It usually works fine but when it has to send more than usual emails (aprox. 25.000 emails), it throws an Exception repeatedly:

Unable to send mail. mail():
Could not execute mail delivery program '/usr/sbin/sendmail -t -i '

The strange thing is that sendmail works properly in other processes such the web server, and sendmail is called in the same way in PHP (Zend). Sendmail only fails in the PHP bulk mail script when a lot emails have been already sent without errors. When the first exception is thrown, next calls to sendmail fail also. It seems as if some queue limit has been reached, but only for this process!

Code of PHP script

The PHP bulk mail scipt main loop is executed thousands of times. In each loop pass calls sendMail with a different $email and $user:

// Sometimes, hundred thousands iterations
foreach($notifications as $j => $notification){
    ...
    $mail->setNotification($notification);
    $this->sendMail($mail, $user);
    ...          
}

$this->sendmail($mail, $user) calls Zend internal method for sending mail. It calls PHP native method mail.

/**
 * Send mail using PHP native mail()
 *
 * @access public
 * @return void
 * @throws Zend_Mail_Transport_Exception if parameters is set
 *         but not a string
 * @throws Zend_Mail_Transport_Exception on mail() failure
 */
public function _sendMail()
{
    ...

        set_error_handler(array($this, '_handleMailErrors'));

        // CALL TO MAIL PHP NATIVE METHOD
        $result = mail(
            $this->recipients,
            $this->_mail->getSubject(),
            $this->body,
            $this->header,
            $this->parameters);
        restore_error_handler();
    }

    if ($this->_errstr !== null || !$result) {
        /**
         * @see Zend_Mail_Transport_Exception
         */
        require_once 'Zend/Mail/Transport/Exception.php';

        // HERE THE EXCEPTION IS THROWN
        throw new Zend_Mail_Transport_Exception('Unable to send mail. ' . $this->_errstr);
    }
}

Proccesses sendmail running

See the ps -aux | grep sendmail output when the bulk mail scipt is working fine

$ ps -aux | grep sendmail
root      6756  0.0  0.0  62240  2468 ?        Ss   18:19   0:08 sendmail: MTA: accepting connections          
root     25766  0.0  0.0  62668  3536 ?        Ss   22:43   0:00 sendmail: MTA: ./r17Lh1fX025764 eml4.in.gr.: client DATA status
root     30978  0.0  0.0  62460  2876 ?        Ss   22:46   0:00 sendmail: MTA: ./r17Lk8li030976 s1.m1r3.onet.pl.: user open
root     31462  0.0  0.0  62672  3536 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkSIg031460 mx2.hotmail.com.: client DATA status
root     31474  0.0  0.0  62672  3540 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkT54031472 mx2.hotmail.com.: client DATA status
root     31494  0.0  0.0  62668  4404 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkUXC031492 gmail-smtp-in.l.google.com.: client RCPT
root     31498  0.0  0.0  62668  3536 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkUn1031496 mx4.hotmail.com.: client DATA status
root     31502  0.0  0.0  62672  3536 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkUti031500 mx3.hotmail.com.: client DATA status
root     31506  0.0  0.0  62672  3500 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkUHw031504 mx4.hotmail.com.: client RCPT
root     31510  0.0  0.0  62672  3496 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkUth031508 mx3.hotmail.com.: client MAIL
root     31514  0.0  0.0  62668  4436 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkVPb031512 gmail-smtp-in.l.google.com.: client DATA status
root     31518  0.0  0.0  62460  2888 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkV9o031516 mx1.hotmail.com.: client EHLO
root     31522  0.0  0.0  62668  4404 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkVD4031520 gmail-smtp-in.l.google.com.: client RCPT
root     31526  0.0  0.0  62460  2852 ?        Ss   22:46   0:00 sendmail: MTA: ./r17LkVcF031524 mx3.hotmail.com.: user open

When the script starts throwing exceptions, ps -aux | grep sendmail outputs almost empty, as expected

$ ps -aux | grep sendmail
root      6756  0.0  0.0  62240  2468 ?        Ss   Feb07   0:49 sendmail: MTA: accepting connections

Some questions

I am quite newbie with sendmail so I appreciate any kind of help. If you need more info, please tell me.

  • Briefly, how does sendmail works for sending mail?
  • Does sendmail have a limit of mails to send (our server is dedicated and has not specific restrictions set by the hosting provider)?
  • Does sendmail have any queue for process that limits the number of mails that can be sent by a given process?
  • Million dollar question: why is the exception beeing thrown and how to solve it? :-)

Thanks in advance for your help!

Community
  • 1
Emilio Nicolás
  • 161
  • 2
  • 8

3 Answers3

3

When you call mail() in PHP on a Unix based system, it forks the sendmail command and passes each recipient as an argument on the command line. However there's a limit to the length of the arguments you can actually pass on the command line! It's a very large limit, 128KiB by default on Linux systems, but 25,000 email addresses vastly exceeds it.

To work around this issue, send your mail in smaller batches, e.g. 1,000 recipients at a time. You should find it trivial to split your array of recipients into groups of 1,000 and loop through them, but if not, visit our sister site Stack Overflow for programming help.

However, your error message seems to indicate that you didn't pass any recipients to the message. So you should perhaps look for a programming error in which you called mail() with no recipients or with invalid recipients.

Community
  • 1
Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • Thanks for your answer. As you can see the code, `mail()` is called in each loop pass with a different email and recipient, it is NOT called only once with all the recipients. – Emilio Nicolás Mar 05 '13 at 12:57
  • Actually, that's **not** apparent from the code - you're using the variable `recipents` which makes it look like it's many recipients, not one. There's nothing in the code you posted that shows how it's called or with what arguments. – Jenny D Mar 05 '13 at 13:05
  • okay, sorry if I was confusing – Emilio Nicolás Mar 05 '13 at 13:35
  • The way to work around silly limitations of the command line is to talk SMTP directly to your neighborhood MTA... – vonbrand Mar 05 '13 at 13:59
  • To work around this he has to implement a queue. He has to place these mail addresses in a queue from where a *separate* background process takes chunks of email addresses and mails them the appropriate message. – adamo Mar 05 '13 at 14:54
  • Thanks for the comments! @adamo why do you think I need to create my own queue? I also think we will end up implementing it. However, I would like to know where the problem is to be sure if I really need to implement it now. – Emilio Nicolás Mar 06 '13 at 09:36
  • @vonbrand I don't think the limitation is on the command line. Why do you think so? – Emilio Nicolás Mar 06 '13 at 09:37
  • @EmilioNicolás, seems I misunderstood an earlier comment. No expert in PHP, sorry. – vonbrand Mar 06 '13 at 09:39
  • @EmilioNicolás Because you cannot put 25000 mail addresses in the command line, therefore you have to break them into pieces and mail one piece at a time. There is your queue. – adamo Mar 06 '13 at 09:44
  • Why *wouldn't* you implement a queue process? It seems like an efficient way of working for this kind of problem. It's how the SMTP server will be processing your emails after all. – Rob Moir Mar 06 '13 at 10:07
  • 1
    In any event, you _should_ be implementing [VERP](https://en.wikipedia.org/wiki/Variable_envelope_return_path) which, as a side effect, will send exactly one recipient at a time to sendmail, solving the problem. – Michael Hampton Mar 06 '13 at 10:09
  • @adamo I have rewrite a bit the question for beeing more understandable. I don't put 25000 mail addresses in the command line. I send one different mail to a diferent user each time in the loop. I don't think the command line limit is the problem. – Emilio Nicolás Mar 06 '13 at 10:27
  • @MichaelHampton thanks for your interest comment. We will take a look to VERP. – Emilio Nicolás Mar 06 '13 at 10:53
0

When you're running this script, are you running it as a different user than the one the web server mailing scripts are run as? It could be something as simple as ulimit for the user running the script.

Jenny D
  • 27,780
  • 21
  • 75
  • 114
  • You are right, the script is running as different user that the one for the web server. Nevertheless, ulimit is unlimited for this user. Besides, UNIX sendmail is running as root. Check `ps -aux` output of one sendmail process: `root 31526 0.0 0.0 62460 2852 ? Ss 22:46 0:00 sendmail: MTA: ./r17LkVcF031524 mx3.hotmail.com.: user open` – Emilio Nicolás Mar 05 '13 at 13:50
  • Yes, **sendmail** is running as root. But the script **calling** sendmail to inject a new message isn't. And "unlimited" still leaves the problem of e.g. too many open file descriptors, lack of sufficient memory or disk space, etc. – Jenny D Mar 05 '13 at 14:01
  • Thanks for your answer! Resources look fine when exceptions start throwing. What would you recommend me to do to ensure lack of resources with ulimit is not the problem? – Emilio Nicolás Mar 06 '13 at 09:21
0

Any mail system is overwhelmed my too many mail messages in short succession. Perhaps you should stop for a little while after sending a batch of messages (say each 100 or so).

In any case, you should review what you are doing here carefully. I very much doubt you have thousands of messages important to the reciepient to send out at the same time. Such behaviour will only get you into the deepest recesses of all email blacklists known to humanity.

vonbrand
  • 1,149
  • 2
  • 8
  • 16
  • Thanks for your answer! The emails are personalized for each user with activity of the web. We have a lot of users and activity. How many emails do you recommend us to send max per unit of time? – Emilio Nicolás Mar 05 '13 at 16:47
  • I don't know, that would depend on your setup. Once I had to send out around a hundred emails (I was program chair, had to send out a batch of acceptance/rejection mails). All was running on my underpowered desktop machine, and I really didn't look farther than waiting a bit after each mail when it choked. – vonbrand Mar 05 '13 at 16:58