Note - I've actually already solved this, but I don't see any detailed write-ups online about this issue so I'll go ahead and ask the question and answer it myself.
I run a service that authenticates as SMTPRelayUser that started failing ADFS authentication about 95% of the time. Inexplicably, this user would very occasionally succeed in authenticating and send all the mail that had queued up over time. In the ADFS Event Viewer logs, I was seeing two errors -- Event 300 and Event 413. Event ID 300 offered up no useful information, but 413 notified me of a strange exception.
Microsoft.IdentityServer.Service.AccountPolicy.ADAccountLookupException: Exception of type 'Microsoft.IdentityServer.Service.AccountPolicy.ADAccountLookupException' was thrown.
What's going on?