Xcode Server Bot not sending email reports

Question

I am running Xcode Server Bots on my local iMac but the email notifications are not being sent.

I have followed many tutorials and have postfix setup and working. I can successfully send emails using the command

sudo echo "Test sending email from Postfix" | mail -s "Test Postfix" xxxxxx@me.com

and my email is received.

Under Xcode > Preferences > Server & Bots > Mail I have left everything blank as per all the tutorials.

For my Bot, under Triggers I have a Periodic Email Report with Schedule set to After each integration and set my from, reply-to and To fields and have every Notify On ticked.

I monitor the smtp and smtp process using the command

sudo log stream --predicate  '(process == "smtpd") || (process == "smtp")' --info

and when I send an email using the above test, I get a log line showing.

However when my Xcode Bot completes, I get absolutely nothing. The Bot logs show

Feb  2 13:47:16  [596] <Info>: Executing trigger 'Periodic Email Report'
Feb  2 13:47:16  [596] <Info>: Executing trigger 'New Issue Email'

So the email report is not triggering any kind of smtp. Is it even using postfix? Can I log something else to see whether it is hitting postfix at all?

EDIT ------

I have found some sendmail logs that seem to show Xcode doesn't have the right permission to use postdrop:

kernel    Sandbox: sendmail(48548) deny(1) forbidden-exec-sugid
sendmail    fatal: execvp /usr/sbin/postdrop: Operation not permitted
sendmail    warning: command "/usr/sbin/postdrop -r" exited with status 1
sendmail    fatal: xcode@<redacted>.com(450): unable to execute /usr/sbin/postdrop -r: Undefined error: 0

My postdrop permissions seem to be correct compared to what I can find online:

-rwxr-xr-x    1 root   wheel        570448  1 Jan  2020 postalias
-rwxr-xr-x    1 root   wheel        466416  1 Jan  2020 postcat
-rwxr-xr-x    1 root   wheel        823488  1 Jan  2020 postconf
-rwxr-sr-x    1 root   _postdrop    588624  1 Jan  2020 postdrop
-rwxr-xr-x    1 root   wheel        431904  1 Jan  2020 postfix
-rwxr-xr-x    1 root   wheel        432032  1 Jan  2020 postkick
-rwxr-xr-x    1 root   wheel        449424  1 Jan  2020 postlock
-rwxr-xr-x    1 root   wheel        431872  1 Jan  2020 postlog
-rwxr-xr-x    1 root   wheel        569632  1 Jan  2020 postmap
-rwxr-xr-x    1 root   wheel        465872  1 Jan  2020 postmulti
-rwxr-sr-x    1 root   _postdrop    588448  1 Jan  2020 postqueue
-rwxr-xr-x    1 root   wheel        466784  1 Jan  2020 postsuper

I have the _postdrop user in my /etc/group file as:

_postdrop:*:28:

Or am I reading this wrong? Does sendmail user need permission to use postdrop?

EDIT 2 -------

It seems this might be an issue with Big Sur making the root partition read only. sudo postfix set-permissions can't do it's thing.

So I guess the question now is, how to install/use a different method for Xcode Server to send emails?

[This](https://developer.apple.com/forums/thread/19809?answerId=292480022#292480022) would be helpful to sending email report triggers. — Raksha Saini, Feb 19 '21 at 02:43
Unfortunately not. I confirm I can send mail from the command line and it shows the `smtp` log, however Xcode log says it's Executing the email trigger, but nothing passes through the smtp at all. — Darren, Feb 19 '21 at 09:22
@zeytin yes. Although I can't accept it as a correct answer, I shall award you the bounty for your effort. I have today found a way to send an email from Xcode and will post it soon after some more testing. — Darren, Feb 21 '21 at 12:13

score 3 · Accepted Answer · answered Feb 21 '21 at 15:50

After a lot of research, asking questions elsewhere and trial and error it seems it's just not going to work. Somewhere along the line, Xcode server doesn't have the right permissions to send email automatically. This could possibly be a bug in Xcode Server, or even Big Sur.

sendmail in itself is quite a beast to setup, but if you have got it working from the command line as suggested in many tutorials, then there is a way you can get emails from Xcode server.

After the suggestion from a member on the Apple forums, it seems that using a Post Integration Script you can in fact call sendmail directly. So i've come up with the following scripts:

Add this as a Post Integration Script: changing the path to where you save the 2 files below.

#!/bin/sh

SCRIPT="/Users/darren/Desktop/mail.sh"
echo "Executing script $SCRIPT"
source $SCRIPT

email.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-GB">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <title>%%XCS_BOT_NAME%%</title>
  <meta name="viewport" content="width=device-width, initial-scale=1.0"/>

  <style type="text/css">
    a[x-apple-data-detectors] {color: inherit !important;}
  </style>

</head>
<body style="margin: 0; padding: 0;">
  <table role="presentation" border="0" cellpadding="0" cellspacing="0" width="100%">
    <tr>
      <td style="padding: 20px 0 30px 0;">

        <table align="center" border="0" cellpadding="0" cellspacing="0" width="600" style="border-collapse: collapse; border: 1px solid #cccccc;">
          <tr>
            <td align="center" bgcolor="%%COLOUR%%" style="padding: 40px 0 30px 0;">
              Integration #%%XCS_INTEGRATION_NUMBER%% of %%XCS_BOT_NAME%% Bot %%RESULT%%
            </td>
          </tr>
          <tr>
            <td bgcolor="#ffffff" style="padding: 40px 30px 40px 30px;">
              <table border="0" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse;">
                <tr>
                  <td style="color: #153643; font-family: Arial, sans-serif;">
                    <h1 style="font-size: 24px; margin: 0;">Latest commit</h1>
                  </td>
                </tr>
                <tr>
                  <td style="color: #153643; font-family: Arial, sans-serif; font-size: 16px; line-height: 24px; padding: 20px 0 30px 0;">
                    <p style="margin: 0;">%%LATESTMERGE%%</p>
                  </td>
                </tr>
              </table>
            </td>
          </tr>
        </table>
      </td>
    </tr>
  </table>
</body>
</html>

mail.sh Edit the 4 variables as needed. You might need to chmod +x mail.sh this file.

#!/bin/sh

# Customise these fields
TO_EMAIL="youremail@me.com"
FROM_EMAIL="xcode@whatever_you_setup_in_sendmail.com"
EMAILTEMPLATE=$(</Users/darren/Desktop/email.html) # $(<path/to/email.html) don't forget the < before the path
EMAIL_AFTER_EVERY_INTEGRATION=1 # true=1/false=0 | 0 will only email on failures/warnings

# Result colour
COLOUR='#888881'
if [[ $XCS_INTEGRATION_RESULT == 'warnings' ]] || [[ $XCS_INTEGRATION_RESULT == 'analyzer-warnings' ]]; then
    COLOUR='#D9CE1F'
elif [[ $XCS_INTEGRATION_RESULT == 'test-failures' ]] || [[ $XCS_INTEGRATION_RESULT == 'build-errors' ]]; then
    COLOUR='#C03636'
elif [[ $XCS_INTEGRATION_RESULT == 'succeeded' ]]; then
    if [ $EMAIL_AFTER_EVERY_INTEGRATION == 0 ]; then
        exit 0
    fi
    COLOUR='#83BC39'
elif [[ $XCS_INTEGRATION_RESULT == 'canceled' ]]; then
    COLOUR='#888881'
fi

# Result
RESULT="Unknown result ${XCS_INTEGRATION_RESULT}"
if [[ ${XCS_INTEGRATION_RESULT} == "succeeded" ]]; then
    RESULT="Succeeded"
elif [[ ${XCS_INTEGRATION_RESULT} == "canceled" ]]; then
    RESULT="was Cancelled"
elif [[ ${XCS_INTEGRATION_RESULT} == "warnings" ]]; then
    if [[ ${XCS_WARNING_COUNT} == 1 ]]; then
        RESULT="Has 1 Warning"
    else
        RESULT="Has ${XCS_WARNING_COUNT} Warnings"
    fi
elif [[ ${XCS_INTEGRATION_RESULT} == "analyzer-warnings" ]]; then
    if [[ ${XCS_ANALYZER_WARNING_COUNT} == 1 ]]; then
        RESULT="Has 1 Warning"
    else
        RESULT="Has ${XCS_ANALYZER_WARNING_COUNT} Warnings"
    fi
elif [[ ${XCS_INTEGRATION_RESULT} == "test-failures" ]]; then
    if [[ ${XCS_TEST_FAILURE_COUNT} == 1 ]]; then
        RESULT="Failed 1 Test"
    else
        RESULT="Failed ${XCS_TEST_FAILURE_COUNT} Tests"
    fi
elif [[ ${XCS_INTEGRATION_RESULT} == "build-errors" ]]; then
    if [[ ${XCS_ERROR_COUNT} == 1 ]]; then
        RESULT="Failed with 1 Error"
    else
        RESULT="Failed with ${XCS_ERROR_COUNT} Errors"
    fi
fi

SUBJECT="$RESULT"

# If not null/empty
if [ -n "${XCS_PRIMARY_REPO_DIR}" ]; then
    LATESTMERGE="$(cd $XCS_PRIMARY_REPO_DIR && git log -1)"
else
    LATESTMERGE="Invalid repo $XCS_PRIMARY_REPO_DIR"
fi

# Replace contents of email
EMAILTEMPLATE="${EMAILTEMPLATE//%%COLOUR%%/$COLOUR}"
EMAILTEMPLATE="${EMAILTEMPLATE//%%XCS_BOT_NAME%%/${XCS_BOT_NAME}}"
EMAILTEMPLATE="${EMAILTEMPLATE//%%XCS_INTEGRATION_NUMBER%%/${XCS_INTEGRATION_NUMBER}}"
EMAILTEMPLATE="${EMAILTEMPLATE//%%RESULT%%/$RESULT}"
EMAILTEMPLATE="${EMAILTEMPLATE//%%LATESTMERGE%%/$LATESTMERGE}"

# echo $EMAILTEMPLATE

echo "Sending report to $TO_EMAIL"

# Send email directly through sendmail
(  
echo "From: $FROM_EMAIL"  
echo "To: $TO_EMAIL"  
echo "MIME-Version: 1.0"  
echo "Subject: $SUBJECT"   
echo "Content-Type: text/html"   
echo $EMAILTEMPLATE  
) | /usr/sbin/sendmail -t

You will now get emails like these:

If anyone knows he we can add the actual failure reasons or errors that would be awesome. There's a list of variables we can use here and I think it might involver looking at the actual build paths.

great stuff, Darren! Thank you for doing the research and sharing it! I wish apple was a bit more attentive to their dev tools :) — Eugene Gordin, Apr 28 '21 at 23:36

zeytin · Answer 2 · 2021-02-20T18:41:01.337

2

Your process is completely true but just for one exception. As you pointed and log said, There is a problem with permissions.

It would be nice sharing by rpm -qV postfix or postfix check full log to figure out the problem deeply but you have already some log which declared Xcode have no permission.

Also you can use your email bot to figure out "Manage Bots in the Xcode Report Navigator". Here is the Apple Document.

I think this will work to set the permissions:

sudo mkdir -p /Library/Server/Mail/Data/Spool
sudo /usr/sbin/postfix set-permissions
sudo /usr/sbin/postfix start

After run this to clear postprop for double check

$ sudo killall postdrop

EDIT1: Or it will be good idea try this commands if not still work

sudo /etc/init.d/postfix stop
sudo killall -9 postdrop

sets the rights and permissions:

sudo chown postfix:postdrop /var/spool/postfix/maildrop
sudo chmod 1730 /var/spool/postfix/maildrop

restart postfix:

sudo /etc/init.d/postfix start

At the final stage, it is supposed to be get an email like that :

FINAL EDIT:

My first and Edit1 Answer doesn't work for PO.

There are some old threads indicate that could be a bug for different MacOS versions. It seems an issue with Big Sur in terms of postfix.

Thread1

Thread2

edited Feb 20 '21 at 18:41

answered Feb 19 '21 at 13:14

zeytin

5,545
4
14
38

Thanks for the help. It seems I am having trouble with the `set-permissions` command. It gives the error `chown: /usr/libexec/postfix: Operation not permitted` I have tried in safe mode too. – Darren Feb 19 '21 at 13:38
You are welcome. I found this one and edited my answer: https://unix.stackexchange.com/questions/471821/how-to-fix-postfix-set-permissions-without-user-named-root – zeytin Feb 19 '21 at 13:47
i am afraid, this is going to linux domain step by step :( https://linux.m2osw.com/snapwebsites-postfixpostdrop18189-warning-mailqueueenter-create-file-maildrop25937318189-permission – zeytin Feb 19 '21 at 13:53
1

I think this might have something to do with Big Sur. I found this https://discussions.apple.com/thread/7933518 to fix the `Operation not permitted` issue. I've disabled `SIP` but now I get the error `chown: /usr/libexec/postfix: Read-only file system` even when I try to manually change the permissions on the `post drop` file. – Darren Feb 19 '21 at 14:01
1

Big Sur doesn't allow modifying the root filesystem. I have followed steps to mount it https://twitter.com/EBADTWEET/status/1275454103900971012 but still get the `Operation not permitted` error when running `set-permissions` – Darren Feb 19 '21 at 14:26
I agree with you. I read the link and it seems a bug. As I am doing same process just as you did in my mac and it is working. Catalina 10.15.6, Xcode 12.2. – zeytin Feb 19 '21 at 14:26
I think it's Big Sur added security rather than a bug. I guess `postfix` needs a different way to handle permissions. – Darren Feb 19 '21 at 14:27
You already changed csrutil authenticated-root status right ? – zeytin Feb 19 '21 at 14:33
Yes. I think it needs a version of post/sendmail that doesn't reside in `usr/bin/` – Darren Feb 19 '21 at 14:37
1

You right, I am going to follow the question, hope you find the exact and short solution. – zeytin Feb 19 '21 at 15:01

score 1 · Answer 3 · answered Feb 18 '21 at 21:55

I'm not too familiar with Xcode, but the problem as you describe it clearly seems to be the communication between the Xcode application and your Postfix SMTP server. I would try the following:

Obtain Xcode logs.
- The post I found below states this can be done by (a) using Console.app, (b) starting Xcode via Terminal.app, or (c) checking for Diagnostic Reports
- https://stackoverflow.com/a/46674772/15238212
Obtain Postfix logs.
- Again, these can be found in Console.app as they should be created by ASL (i.e. macOS's syslog)
- Additionally, you could specify your own log file as stated in the documentation on the Postfix website: http://www.postfix.org/MAILLOG_README.html

Hopefully by doing this you be able to see if your configured mail updates are making it out of Xcode and reaching your Postfix server.

Thank you for your help. I have managed to find the logs and it seems it might be a permission issue. I've attached the logs to my question above. Thx — Darren, Feb 19 '21 at 10:52

Xcode Server Bot not sending email reports

3 Answers3