Unable to deliver email to hotmail or gmail

Question

I have been attempting to send emails to registered users of my application via php.

I am using LAMP on debian ( digital ocean droplet ). Sendmail and opendkim.

I have set up SPF, DKIM and DMARC and confirmed these are working correctly through third party web based services.

I got a 100% score on this service with no warnings: http://www.mail-tester.com/

On gmail the emails arrive as spam, on hotmail they don't arrive at all, although I get a notification of a bounced email not being accepted on my end several days later.

Here is the source of an email received by hotmail after adding the sender to my safe senders list.

x-store-info:qAUQJzZ73IJCLUJ+0n7ZQ0tyh3aLbvsRShq0lkPgv3IVOooErkSkewDsP+t1Cax/muSI9UyoB4MPpzTF7SmsHoXotERao0AMdxy/dOy2I80PlEnXiwFP/Ayeh8hnqX+UkFVWr84Ulqk=
Authentication-Results: hotmail.com; spf=pass (sender IP is <ip>; identity alignment result is pass and alignment mode is relaxed) smtp.mailfrom=challenge@<domain>; dkim=pass (identity alignment result is pass and alignment mode is relaxed) header.d=challenge.<domain>; x-hmca=pass header.id=challenge@<domain>
X-SID-PRA: challenge@<domain>
X-AUTH-Result: PASS
X-SID-Result: PASS
X-Message-Status: n:n
X-Message-Delivery: Vj0xLjE7dXM9MDtsPTA7YT0wO0Q9MztHRD0zO1NDTD02
X-Message-Info: 6YO/4nwP5t1mtfSNVgWW2U47Sdk9xX1SAWUsoLSyWydDzdrrmeugrkjPJG7agB5LIyETvhvrEb6Xr8XvqMsKUN2MMCgO2BvUkt24wTtZLl+hft6A9mG1JLiWYRqqeuRiFrKOc4kGXDyuvt8lhEDkfSC0zQx6CevwL2OlyGkB+7DFMUuZiMI2/eFVv4653QhUysdXIVVT0Jqr/wP5LJ2cP+YNjWLC9Kmx
Received: from challenge.<domain> ([<ip address>]) by BAY004-MC3F17.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008);
Wed, 8 Apr 2015 14:27:38 -0700
Received: from challenge.<domain> (localhost [127.0.0.1])
by challenge.<domain> (8.14.4/8.14.4/Debian-4) with ESMTP id t38LRc1D027844
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT)
for <henrypenny@hotmail.com>; Wed, 8 Apr 2015 21:27:38 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;
d=challenge.<domain>; s=default; t=1428528458;
bh=fdkeB/A0FkbVP2k4J4pNPoeWH6vqBm9+b0C3OY87Cw8=;
h=Date:From:To:Subject:From;
b=PdpGBHg7bA1RKI1lGU36jCbY+/IRebFtCHQlYZvbu2s5TV5gb+/sCG9fVjybaUPKI
xN+6PC58F3V+EpPtmFVddbpfyanMy1Rs/acFrNDSZLM5XeggWN4mLxQvo48iCJxOs7
crERNaCdhU+D3tDfUmbdPfBXnP89ql9lEopiuzis=
Received: (from devops@localhost)
by <domain> (8.14.4/8.14.4/Submit) id t38LRbio027843
for henrypenny@hotmail.com; Wed, 8 Apr 2015 21:27:37 GMT
Date: Wed, 08 Apr 2015 21:27:37 +0000
From: challenge@<domain>
To: henrypenny@hotmail.com
Subject: Test from PTI server
Message-ID: <55259d49.3vDGyJdimZF+gCyx%challenge@<domain>>
User-Agent: Heirloom mailx 12.5 6/20/10
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Return-Path: challenge@challenge.<domain>
X-OriginalArrivalTime: 08 Apr 2015 21:27:38.0574 (UTC) FILETIME=[D67E16E0:01D07242]

Test

I have successfully sent emails from a different server using the same sender and they get straight through gmail and hotmail. So I have to conclude that my DNS and SPF set up is fine. Ironically the other server has no DKIM or DMARC. (note: the other server has postfix)

I'm a bit confused by the multiple received headers.

Received: from challenge.<domain>...
Received: from challenge.<domain>...
Received: (from devops@localhost)
by challenge.<domain>...

I've checked the IP address against http://www.barracudacentral.org/lookups

The IP address 104.236.167.229 is not currently listed as "poor" on the Barracuda Reputation System.

http://mxtoolbox.com/ shows the domain is on no blacklists.

http://www.dnsbl.info/ is completely clear too.

Here is the mail log:

Apr  8 23:27:47 challenge sendmail[28470]: t38NRlEL028470: from=challenge@<domain>, size=346, class=0, nrcpts=1, msgid=<5525b973.KWFuUvRrPkQR+tEN%challenge@<domain>>, relay=devops@localhost
Apr  8 23:27:47 challenge sendmail[28470]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-GCM-SHA384, bits=256/256
Apr  8 23:27:47 challenge sm-mta[28471]: STARTTLS=server, relay=localhost [127.0.0.1], version=TLSv1/SSLv3, verify=NOT, cipher=DHE-RSA-AES256-GCM-SHA384, bits=256/256
Apr  8 23:27:47 challenge sm-mta[28471]: t38NRlgr028471: from=<challenge@<domain>>, size=511, class=0, nrcpts=1, msgid=<5525b973.KWFuUvRrPkQR+tEN%challenge@<domain>>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Apr  8 23:27:47 challenge opendkim[24594]: t38NRlgr028471: DKIM-Signature header added (s=default, d=<domain>)
Apr  8 23:27:47 challenge sm-mta[28471]: t38NRlgr028471: Milter insert (1): header: DKIM-Signature:  v=1; a=rsa-sha256; c=relaxed/simple;\n\td=<domain>; s=default; t=1428535667;\n\tbh=g3zLYH4xKxcPrHOD18z9YfpQcnk/GaJedfustWU5uGs=;\n\th=Date:From:To:Subject:From;\n\tb=ajpBs8Y8C8oM4maqGW5ZAwdQY/mJi+p0z3tIgyRSTjNelFqRvy4O5pvOOH8N/tN0Y\n\t j8pqP32gURz57Mhpxh1HeWsZ0nyGn17y3/uKA7Kek6fBqxA+Zx0Ucyujl7URiSvKC9\n\t q3rtTcxYd562Y2zsYA/cearagZ/9uuX93PxZdyaM=
Apr  8 23:27:47 challenge sendmail[28470]: t38NRlEL028470: to=henrypenny@hotmail.com, delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30346, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (t38NRlgr028471 Message accepted for delivery)
Apr  8 23:27:47 challenge sm-mta[28473]: STARTTLS=client, relay=mx4.hotmail.com., version=TLSv1/SSLv3, verify=OK, cipher=ECDHE-RSA-AES256-SHA384, bits=256/256
Apr  8 23:27:48 challenge sm-mta[28473]: t38NRlgr028471: to=<henrypenny@hotmail.com>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=120511, relay=mx4.hotmail.com. [65.55.33.119], dsn=2.0.0, stat=Sent ( <5525b973.KWFuUvRrPkQR+tEN%challenge@<domain>> Queued mail for delivery)

I've also setup the reverse DNS lookup and tested it.

I get this message when I run sendmailconfig:

Creating /etc/mail/databases...

Checking filesystem, this may take some time - it will not hang!
  ...   Done.

Checking for installed MDAs...
Creating /etc/mail/sasl/sasl.m4...

Ah, you're setup with SASL2 !

Unfortunately, there is no automagic way to migrate to /etc/sasldb2 :(

You'll want to make sure /etc/default/saslauthd is setup to start,
and has at least MECHANISMS="pam" !

If you find out what more is needed, please let me know!

Creating/Updating SSL(for TLS) information
Creating /etc/mail/tls/starttls.m4...
You already have sendmail certificates

Checking {sendmail,submit}.mc and related databases...

Answer 1

There are other factors that come in place other than your source email address and DNS and SPF settings, including the reputation of the IP address itself and the type of IP address. Many mail services reject or downvote emails that are being recieved from IP addresses in a dynamic pool or are in a cloud service.

Look up the following for clues on if you have an IP issue.

• DNSBL
• http://www.barracudacentral.org/lookups
• http://mxtoolbox.com/blacklists.aspx

Assuming the IP is not an issue, since you are getting a bounce days after the attempted delivery, I suspect that the email might be stuck in your server while your server is retrying. If that is the case, check the Sendmail logs to see what happens. Note that you may need to increase the verbosity of Sendmail logs in /etc/mail/sendmail.cf

Also note that the issue might be because of reverse DNS checks that the receiving email server might be doing. For digital ocean droplet, the default setup usually causes the reverse DNS check to fail, and this would show up in the logs. If I recall correctly for the droplets, you would need to use the DNS control panel and update your PTR record by setting the proper hostname in the control panel, such that the droplets hostname is the domain name resulting in the reverse DNS to be valid.