First, a mail client (usually a mail server) queries a DNS server to determine the domain’s MX record so it knows where to send the mail.
A mail client could also query the DNS server to determine if a MTA-STS record exists as well.
If the record exists, then the server connects to a predetermined URL (
mta-sts.<domain>/.well-known/mta-sts.txt) and obtains the rule that has been established by the system owner.
This rule is cached by the system and doesn’t change unless the DNS record’s ID changes so, like HSTS, once the server connects the first time, it knows that it should always connect to that domain’s SMTP server(s) using encryption.
If there is a downgrade attack, or some other problem where encryption is not available, the connection fails and mail is not sent.
It should be noted that the DNS records should/must be secured with DNSSEC for this whole thing to work well.