If mails are sent but don’t get through, that is usually due to sender IPs being blocked or something like that. In search of a way for debugging I found this line
log stream --predicate '(process == "smtpd") || (process == "smtp")' --info
which spits out smtp infos. Maybe you can run this from the command line on your server.
Sending mail is much more reliable if you use a provider to send your mail through.