- Verified the 'Immediate Alert' job was running every five minutes successfully.
- Double checked the config information for outgoing email in Central Admin.
- Verified the SMTP server was configured to allow emails from the Web and App server.
- ULS was not helpful
- Restarted SharePoint Timer service
- Restarted IIS
- Even restarted all servers in the farm
Before enabling verbose logging and slogging through all that data, I decided to broaden my testing scope and I created a simple workflow that sent me an email upon trigger. The workflow completed with an error "The e-mail message cannot be sent. Make sure the outgoing e-mail settings for the server are configured correctly. For more information, please read this article: http://go.microsoft.com/fwlink/?LinkID=323543&clcid=0x409"
This information was not immediately helpful but it did confirm that the issue was within the SharePoint farm and not with Exchange or some other outside service.
Taking another look at Outgoing E-mail Settings, I see one setting that is available for SharePoint 2016 that was not available in SharePoint 2010; "Use TLS connection encryption" which is defaulted to 'Yes'. I Set it to 'No', restarted IIS an the SharePoint Timer service. Resetting IIS and Timer was probably unnecessary but I wanted to be sure.
I was now able to receive alerts and workflow emails. Turns out our email server was not using a compatible version of TLS. It would have been nice to find an error with these details.
So, if you are not receiving alerts and system emails but you are sure your environment is configured correctly, disable TLS and see what happens. If you need encryption, work with your Exchange admin to ensure compatible versions are available.