Send Mail Job hangs in Azure DevOps Server

Question

We are running Azure DevOps Server 2020 Update 1.2

For the second time this month, the email notification system stopped working. The test email function still works though, so the smtp server details are good.

Restarting the background job agent "resolved" the issue, emails in the queue will be released over the next few hours.

As this happened twice now, i want to look into why this might be happening. Is there a query i can run on the database to find a relevant error message?

Some data:

Event log while restarting the job agent

Detailed Message: TF400804: The Azure DevOps Job Agent was terminating execution but the following jobs would not come to a stop in a timely manner: [JobId: b1516502-4633-432b-bdb3-74c802c5f2b7, 
Name: Team Foundation Server Send Mail Job, Extension: 
Microsoft.TeamFoundation.JobService.Extensions.Core.SendMailJobExtension]; [JobId: a96d6177-beef-477a-a2ee-2c31433214d0, 
Name: Notifications E-Mail Delivery, Extension: Microsoft.VisualStudio.Services.Notifications.Extensions.UserDeliveryJob]; [JobId: 8833fc71-42ca-441b-ab12-25314877772d, 
Name: Git User Delivery, Extension: Microsoft.TeamFoundation.JobService.Extensions.Core.UserDeliveryJobGit]; [JobId: 631f49b3-46e1-42ec-8fff-081bd176c18a, 
Name: WorkItem User Delivery, Extension: Microsoft.TeamFoundation.JobService.Extensions.Core.UserDeliveryJobWorkItems]; [JobId: 8833fc71-42ca-441b-ab12-25314877772d, 
Name: Git User Delivery, Extension: Microsoft.TeamFoundation.JobService.Extensions.Core.UserDeliveryJobGit]

Event log - every 30 minutes

Detailed Message: TF400527: An error occurred while attempting to send an email (Request Id: 23eacfe9-3571-487e-ba2b-2faad799988b, Requested By: a10c324c-6462-4808-80f5-ebce59c90074). Further email sending errors that occur within the next 5 minutes might not be logged. Verify that the email notification settings are correct in the Azure DevOps Administration Console.

Exception Message: Failure sending mail. (type SmtpException)
Exception Stack Trace:    at Microsoft.TeamFoundation.Framework.Server.MailSenderExtensions.SendSynchronously(VssMailSender sender, IVssRequestContext requestContext, MailMessage message, String area, String layer, Stopwatch stopwatch)
   at Microsoft.TeamFoundation.Framework.Server.TeamFoundationMailService.Send(IVssRequestContext requestContext, IEnumerable`1 messages)
   at Microsoft.TeamFoundation.JobService.Extensions.Core.SendMailJobExtension.SendMails(IVssRequestContext requestContext, IEnumerable`1 mailsToBeSent, ICollection`1 sentMails)

Inner Exception Details:

Exception Message: The operation has timed out. (type WebException)Exception Stack Trace:    at System.Net.Mail.SmtpConnection.ConnectAndHandshakeAsyncResult.End(IAsyncResult result)
   at System.Net.Mail.SmtpTransport.EndGetConnection(IAsyncResult result)
   at System.Net.Mail.SmtpClient.ConnectCallback(IAsyncResult result)

Is there anything else i can extract from the logs that might be useful?

Either now or the next time it happens

It seems to not being able to open a connection. Any firewall, network issues at the time the job ran? — jessehouwing, Nov 28 '22 at 10:48
Both are on the same internal domain, there should not be a firewall issue. Also, since restarting the job helps, the firewall seems unlikely. Im thinking more in lines of a large email, or something blocking the queue, is there anything in the database that can indicate that? — Nico, Nov 28 '22 at 10:58
hi nico, what is the latest status of your issue, does my answer help? — Ceeno Qi-MSFT, Dec 02 '22 at 12:01
It might, i will test the next time it happens again, thank you for the suggestion! — Nico, Dec 02 '22 at 15:08

score 1 · Answer 1 · answered Nov 29 '22 at 07:51

From TF400527, The exception shows this message: A connection attempt failed because the connected party did not properly respond after a period of time

It looks like TFS is failing to reach your configured SMTP server. Is it possible you have an outbound firewall rule on the TFS server preventing it from accessing your SMTP server? Or that you have an inbound rule on your SMTP server that is preventing connectivity from this TFS instance?

Next time the issue is observed, you could check the table in database: microsoft_sql_server_management_studio -> databases -> azuredevops_{your collection name} -> tables -> dbo.tbl_EventNotification -> right_click_to_select_xxxxxx, you could select to define your report.

From the status column, you could check if the notification event is triggered properly from the Azure DevOps server.

Send Mail Job hangs in Azure DevOps Server

1 Answers1