This page documents the configuration of the mail server. There is no user-facing documentation yet.

Running a mail server can be surprisingly hard, considering how long the protocols have been around. However, I believe it is something that can be managed by any moderately experience administrator, and in fact could be better automated if we could figure out a simple set of standard configurations. Hopefully, this page should cover the latest good configuration for small servers.

  1. DNS
    1. SPF
    2. DKIM
    3. DMARC
  2. Postfix
  3. Postfix opportunistic TLS
  4. MTA-STS
  5. Postfix SASL configuration
    1. Dovecot
    2. Postfix server
    3. Testing
    4. Client configuration
    5. References
  6. Delivery and retrieval over SSH
  7. Spam filtering
    1. DMARC / SPF pre-checks
    2. Todo
  8. Dovecot and mail filters
    1. Todo
  9. Webmail
  10. Mailing lists
    1. Resolved Issues
    2. Remaining issues
    3. Tested
    4. Future work
  11. Disaster recovery
    1. new VM setup
    2. Bootstrap

DNS

One of the trickiest parts of configuring email is to setup DNS correctly. You must have an A record that points to your server's IP address that is always up to date, and a reverse PTR record that points to the same name. Otherwise a lot of servers will bounce your outgoing emails.

Furthermore, some providers block certain IP blocks as being "consumer" addresses that are never supposed to send email. There is little you can do against this but hope for the best or find a provider with "clean addresses".

It is also possible that the address you are given by your provider is blocked because it sent spam in the past: in this case your only hope is to get the address unblocked or have your provider give you another IP address.

SPF

SPF records are fairly simple: they specify policies on the servers allowed to send email for a specific domain. They are an extension of the traditional "reverse DNS must match" kind of policies.

In my case, it was simply a matter of saying that my main server is responsible for all outgoing mail:

@       TXT     "v=spf1 a mx -all"

Breaking it down, this means:

Because the mail exchanger (which receives email) is the same as the outgoing email server, this is sufficient. If those services would be split up (for example with a separate machine sending email like a mailing list server), this would need to be expanded, for example by including a:lists.example.com.

Configurations can be tested with those tools:

DKIM

DKIM is a standard (RFC6376) to sign email headers and prevent forgeries.

First install the DKIM server and tools:

apt install opendkim opendkim-tools

Create some configuration directories:

mkdir -p /etc/opendkim/keys

In /etc/opendkim.conf, we add those lines:

# write socket inside postfix's chroot
Socket                  local:/var/spool/postfix/opendkim/opendkim.sock

# key/host mappings
SigningTable        refile:/etc/opendkim/signing.table
KeyTable        /etc/opendkim/key.table

# Hosts to generate signatures for
InternalHosts       /etc/opendkim/internal.hosts
# Hosts to ignore when verifying signatures, same as above
ExternalIgnoreList  /etc/opendkim/internal.hosts

We would have used Domain if we had only one domain, but since we have many, we need those tables. First one, is signing.table:

*@anarc.at marcos
*@orangeseeds.org marcos

First part is a pattern, second is a key that is then found in key.table:

marcos anarc.at:marcos:/etc/opendkim/keys/marcos.private

First field is the signing.table key. Second is field is a colon-separated list of fields: the domain name, a selector (can be anything, but we picked the hostname), and the private key file.

Then generate the private key and DNS record

opendkim-genkey --directory=/etc/opendkim/keys/ --selector=marcos --domain=anarc.at --verbose

The private key is the .private file specified above, and the DNS record is written in a .txt file. The latter should be included in the zone file:

$INCLUDE "/etc/opendkim/keys/marcos.txt"

Unfortunately, that fails with an obscure permission denied, maybe because it's outside of the normal bind directories and/or apparmor. Instead, copy-paste the content, which will look something like this:

marcos._domainkey       IN      TXT     ( "v=DKIM1; h=sha256; k=rsa; "
          "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAubF5LI+R0lroVTItfbbs714+BW3waB34fUZ7wJT6Vrj3QVNg82bjIAL9u+WMOGt4okNi/QhjtoofqeUTVJycULiu1YXn6yQ8Dqrvm4s4uO8mwErOPV2mIplyRVwLEmS5zCw4UcVTdyDnPHqbzrRXAJfwJ3qFwwLwRraLDuuzqv+RX+mb5/s4VgjAXXAFHzDOdhcj7kGk8CzxXW"
          "ZWt8W7ilJUSoRoslBoA/jj5TMUU/xtbtSV0kZUVf8Y+0IKuxJHTlSrDqJ/PcMJZqMHtRY2ZQPtzaGPeFLJRN1J4U+krqerJiCc+n7P0KBS/yPb0H24mbWGP2WFxou3s3XdYeGSiQIDAQAB" )  ; ----- DKIM key marcos for anarc.at

Then this will tell other servers we always sign our mails, and refuse unsigned emails:

; enforce DKIM on all mails from this domain, deprecated
_adsp._domainkey        IN      TXT     "dkim=all"

Note that the above "ADSP" policy is now discouraged in favor of DMARC.

Fix permissions on the key files:

chmod o-rw /etc/opendkim/keys
chown opendkim:root /etc/opendkim/keys/*

The DKIM server is then restarted and checked:

service opendkim restart
opendkim-testkey -d anarc.at -s marcos -vv

The keys not secure message means you are not using DNSSEC.

Then create the directory with proper permissions:

mkdir /var/spool/postfix/opendkim
chown opendkim /var/spool/postfix/opendkim

Allow postfix to access the opendkim socket:

adduser postfix opendkim

And restart dkim again:

service opendkim restart

Then hook it up in postfix:

postconf -e milter_protocol=6
postconf -e milter_default_action=accept
postconf -e smtpd_milters=local:opendkim/opendkim.sock
postconf -e non_smtpd_milters=local:opendkim/opendkim.sock

If emails are double-signed, add receive_override_options=no_milters to loopback smtpd servers in master.cf, for example:

localhost:10026        inet    n       -       n       -       10      smtpd
    -o smtpd_tls_security_level=none
    -o content_filter=
    -o myhostname=delivery.anarc.at
    -o receive_override_options=no_milters

Then postfix must be reloaded:

postfix reload

Then follow the test procedure, below.

Adding a new domain

Debian.org added support for DKIM in 2020. To configure this on my side, I had to do the following, on top of the above...

  1. add this line to signing.table:

    *@debian.org marcos-debian.anarcat.user
    
  2. add this line to key.table:

    marcos-debian.anarcat.user debian.org:marcos-debian.anarcat.user:/etc/opendkim/keys/marcos-debian.anarcat.user.private
    

    Yes, that's quite a mouthful! That magic selector is long in that way because it needs a special syntax (specifically the .anarcat.user suffix) for Debian to be happy. The -debian string is to tell me where the key is published. The marcos prefix is to remind me where the private is used.

  3. generate the key with:

    opendkim-genkey --directory=/etc/opendkim/keys/ --selector=marcos-debian.anarcat.user --domain=debian.org --verbose
    

    This creates the DNS record in /etc/opendkim/keys/marcos-debian.anarcat.user.txt (alongside the private key in .key).

  4. restart opendkim:

    service opendkim restart
    

    The DNS record will look something like this:

    marcos-debian.anarcat.user._domainkey   IN  TXT ( "v=DKIM1; h=sha256; k=rsa; "
    "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtKzBK2f8vg5yV307WAOatOhypQt3ANQ95iDaewkVehmx42lZ6b4PzA1k5DkIarxjkk+7m6oSpx5H3egrUSLMirUiMGsIb5XVGBPFmKZhDVmC7F5G1SV7SRqqKZYrXTufRRSne1eEtA31xpMP0B32f6v6lkoIZwS07yQ7DDbwA9MHfyb6MkgAvDwNJ45H4cOcdlCt0AnTSVndcl"
    "pci5/2o/oKD05J9hxFTtlEblrhDXWRQR7pmthN8qg4WaNI4WszbB3Or4eBCxhUdvAt2NF9c9eYLQGf0jfRsbOcjSfeus0e2fpsKW7JMvFzX8+O5pWfSpRpdPatOt80yy0eqpm1uQIDAQAB" )  ; ----- DKIM key marcos-debian.anarcat.user for debian.org
    
  5. The "p=MIIB..." string needs to be joined together, without the quotes and the p=, and sent in a signed email to changes@db.debian.org:

    -----BEGIN PGP SIGNED MESSAGE-----
    dkimPubKey: marcos.anarcat.user MIIB[...]
    -----BEGIN PGP SIGNATURE-----
    [...]
    
  6. Wait a few minutes for DNS to propagate. You can check if they have with:

    host -t TXT marcos-debian.anarcat.user._domainkey.debian.org nsp.dnsnode.net

    (nsp.dnsnode.net being one of the NS records of the debian.org zone.)

If all goes well, the tests (below) should pass when sending from your server as anarcat@debian.org.

Testing

Test messages can be sent to dkimvalidator, mail-tester.com, or check-auth@verifier.port25.com. Those tools will run Spamassassin on the received emails and report the results. What you are looking for is:

If one of those is missing, then you are doing something wrong and your "spamminess" score will be worse. The latter is especially tricky as it validates the "Envelope From", which is the MAIL FROM: header as sent by the originating MTA, which you see as from=<> in the postfix lost.

The following will happen anyways, as soon as you have a signature, that's normal:

And this might happen if you have a ADSP record but do not correctly sign the message with a domain field that matches the record:

That's bad and will affect your spam core badly. I fixed that issue by using a wildcard key in the key table:

--- a/opendkim/key.table
+++ b/opendkim/key.table
@@ -1 +1 @@
-marcos anarc.at:marcos:/etc/opendkim/keys/marcos.private
+marcos %:marcos:/etc/opendkim/keys/marcos.private

References used:

Another test tool is https://mxtoolbox.com/emailhealth.

DMARC

I am using this simple, non-restrictive DMARC policy:

_dmarc  IN  TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@anarc.at"

Breaking it down, this means:

I am not clear yet on how that interacts with DKIM and SPF, but that seems like a safe way to start.

The above will lead to reports landing in your mailbox. To parse them, you can use the dmarc-cat tool which I packaged for Debian.

Postfix

I am using the Postfix server. I find it easier to configure than the other common mail server, Exim, which I never bothered learning.

Debian does a good job at configuring Postfix for you, when you install the package. It does ask a bit too many questions for my taste, but it does help with the grunt of the work. Here are, some pointers to the answers you want to give:

There are literally hundreds of configuration settings available in Postfix, the best is to the excellent Postfix documentation for further guides. Some of them are simplified below.

Postfix opportunistic TLS

This configures Postfix to offer a X509 certificate on inbound and outbound SMTP connections but also accept unencrypted connections as a fallback, in main.cf:

# configure certificates
smtpd_tls_cert_file = /etc/letsencrypt/live/anarc.at/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/anarc.at/privkey.pem
smtp_tls_cert_file = ${smtpd_tls_cert_file}
smtp_tls_key_file = ${smtpd_tls_key_file}
# opportunistic encryption
smtpd_tls_security_level = may
smtp_tls_security_level = may

See the TLS_README and smtp_tls_security_level for more information.

This patch is required to disable TLS when a content_filter is configured (see FILTER_README and below).

diff --git a/postfix/master.cf b/postfix/master.cf
index b0ed875..6329c70 100644
--- a/postfix/master.cf
+++ b/postfix/master.cf
@@ -12,7 +12,9 @@
 smtp      inet  n       -       -       -       -       smtpd
         -o content_filter=smtp:127.0.0.1:10025
         -o myhostname=mx.anarc.at
+        -o smtp_tls_security_level=none
 localhost:10026        inet    n       -       n       -       10      smtpd
+        -o smtpd_tls_security_level=none
         -o content_filter=
         -o myhostname=delivery.anarc.at
 submission inet n       -       -       -       -       smtpd

Without this patch, local delivery would hang during my tests.

MTA-STS

The above has now somewhat been standardized as RFC 8461, "SMTP MTA Strict Transport Security".

For me this involved creating the following file in my well-known directory (/var/www/.well-known/mta-sts.txt):

version: STSv1
mode: testing
mx: marcos.anarc.at
max_age: 86400

Then a set of new record needs to be added to DNS:

mta-sts IN CNAME marcos
_smtp._tls     IN      TXT "v=TLSRPTv1;rua=mailto:postmaster@anarc.at"
_mta-sts   300   IN   TXT   "v=STSv1; id=20190225113927Z;"

The id field should be a unique string that changes when there's a policy change. I picked the format output by date +%Y%m%d%H%M%SZ.

Then a vhost needs to be created for the above file to be accessible:

<VirtualHost *:80>
    ServerName mta-sts.anarc.at
    ServerAlias mta-sts.orangeseeds.org
    DocumentRoot /var/www/html/
    #Redirect / https://mta-sts.anarc.at/
</VirtualHost>

<VirtualHost *:443>
    ServerName mta-sts.anarc.at
    ServerAlias mta-sts.orangeseeds.org
    DocumentRoot /var/www/html/
    #Use common-letsencrypt-ssl mta-sts.anarc.at
</VirtualHost>

This needs to be authenticated with certbot of course and uncomment the redirect and Use lines when done:

rndc reload
service apache2 reload
certbot certonly --webroot --webroot-path /var/www/html/ -d mta-sts.anarc.at -d mta-sts.orangeseeds.org

Then the configuration can be checked on aykevl.nl or hardenize.com.

This only works for incoming email. For outgoing email, Postfix needs to be able to check the TLS policy (which it could do with smtp_tls_policy_maps) but the MTA-STS checks are not supported in the daemon itself. Thankfully, there's a third-party daemon called postfix-mta-sts-resolver which can do this. Unfortunately it wasn't in Debian when I worked on this (since then solved, see bug 917366) so I didn't get to deploy that just yet.

References:

Postfix SASL configuration

This allows users to submit mails using their regular IMAP credentials.

Dovecot

We don't want to configure a new authentication system for Postfix, so we delegate to the already existing one, which is Dovecot.

In the default configuration, the following patch is required to Dovecot to enable a control socket exposed to Postfix:

diff --git a/dovecot/conf.d/10-auth.conf b/dovecot/conf.d/10-auth.conf
index 1c59eb4..187b262 100644
--- a/dovecot/conf.d/10-auth.conf
+++ b/dovecot/conf.d/10-auth.conf
@@ -97,7 +97,7 @@
 #   plain login digest-md5 cram-md5 ntlm rpa apop anonymous gssapi otp skey
 #   gss-spnego
 # NOTE: See also disable_plaintext_auth setting.
-auth_mechanisms = plain
+auth_mechanisms = plain login

 ##
 ## Password and user databases
diff --git a/dovecot/conf.d/10-master.conf b/dovecot/conf.d/10-master.conf
index e3d6260..5068100 100644
--- a/dovecot/conf.d/10-master.conf
+++ b/dovecot/conf.d/10-master.conf
@@ -93,9 +93,11 @@ service auth {
   }

   # Postfix smtp-auth
-  #unix_listener /var/spool/postfix/private/auth {
-  #  mode = 0666
-  #}
+  unix_listener /var/spool/postfix/private/auth {
+    mode = 0660
+    user = postfix
+    group = postfix
+  }

   # Auth process is run as this user.
   #user = $default_internal_user

Then notify Dovecot of the change:

service dovecot reload

Postfix server

This enables SASL authentication with Dovecot. It ensures passwords are only sent over a secure channel and censors the original IP address.

--- a/postfix/main.cf
+++ b/postfix/main.cf
@@ -46,8 +46,9 @@ home_mailbox = Maildir/
 #mailbox_command = /usr/bin/procmail -a "$EXTENSION"
 smtpd_recipient_restrictions = reject_unlisted_recipient,
                                permit_mynetworks,
+                               permit_sasl_authenticated,
                                reject_non_fqdn_recipient,
                                reject_unauth_destination,
                                check_policy_service inet:127.0.0.1:10023,
                                reject_rbl_client zen.spamhaus.org

In other words, make sure that permit_sasl_authenticated is added to smtpd_recipient_restrictions.

Then you need to hook Dovecot SASL authentication into Postfix and make sure it is not offered in cleartext:

postconf -e smtpd_sasl_type=dovecot
postconf -e smtpd_sasl_path=private/auth
postconf -e smtpd_tls_auth_only=yes

Then enable the submission port (587) in master.cf:

diff --git a/postfix/master.cf b/postfix/master.cf
index f4f096d..b0ed875 100644
--- a/postfix/master.cf
+++ b/postfix/master.cf
@@ -15,11 +15,11 @@ smtp      inet  n       -       -       -       -       smtpd
 localhost:10026        inet    n       -       n       -       10      smtpd
         -o content_filter=
         -o myhostname=delivery.anarc.at
-#submission inet n       -       -       -       -       smtpd
-#  -o smtpd_tls_security_level=encrypt
-#  -o smtpd_sasl_auth_enable=yes
-#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
-#  -o milter_macro_daemon_name=ORIGINATING
+submission inet n       -       -       -       -       smtpd
+  -o smtpd_tls_security_level=encrypt
+  -o smtpd_sasl_auth_enable=yes
+  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
+  -o milter_macro_daemon_name=ORIGINATING
 smtps     inet  n       -       -       -       -       smtpd
   -o smtpd_tls_wrappermode=yes
 #  -o smtpd_sasl_auth_enable=yes

As an extra, you can protect your users' privacy by hiding their IP address with this:

postconf -e smtpd_sasl_authenticated_header=yes
postconf -e header_checks=regexp:/etc/postfix/header_checks

The following header_checks file does the magic censoring:

/^Received: from (.* \([-._[:alnum:]]+ [[.[:digit:]]{7,15}\]\)).*?(<span class="createlink">:space:</span>+).*\(Authenticated sender: ([^)]+)\).*by ([-._[:alnum:]]+) \(([^)]+)\) with (E?SMTPS?A?) id ([A-F[:digit:]]+).*/
        REPLACE Received: from [127.0.0.1] (localhost [127.0.0.1])$2(Authenticated sender: $3)${2}with $6 id $7

Note that this will expose their usernames, but that is actually useful because it allows you to track down eventual spammers.

Testing

This will try to relay an email through server example.net to the example.com domain using TLS over the submission port (587) with user name anarcat and a prompted password (-ap -pp).

swaks -f anarcat@example.net -t anarcat@example.com -s example.net -tls -p 587 -au anarcat -ap -pp

Client configuration

This should be fairly straightforward on most clients, e.g. Thunderbird should autodetect authentication on port 25, and if that is blocked, port 587 with STARTTLS should be used.

Emacs

The following customization must be performed to send email through the Emacs SMTP library:

(setq send-mail-function (quote smtpmail-send-it))
(setq smtpmail-default-smtp-server "mail.anarc.at")
(setq smtpmail-smtp-service 587)
(setq smtpmail-stream-type (quote starttls))

The first time you will send an email, it will ask you to save your credentials to ~/.authinfo. I accepted, but then went to that file and erased the password that was stored. That way Emacs remembers the password and prompts me as necessary, without storing the precious password on disk. This file is entirely customizable as well through the auth package. For example, you can store the password on disk but encrypt it using GPG, simply by encrypting the file as ~/.authinfo.gpg and removing the original. See the gpg documentation for more information.

A more detailed documentation of my client setup is in 2016-05-12-email-setup.

Postfix

To configure Postfix as a client for the above, the following configuration can be used:

postconf -e smtp_sasl_auth_enable=yes
postconf -e smtp_sasl_password_maps=hash:/etc/postfix/sasl/passwd
postconf -e smtp_sasl_security_options= 
postconf -e relayhost=[hostname]:587
postconf -e smtp_tls_security_level=encrypt
postconf -e smtpd_tls_security_level=encrypt
postmap /etc/postfix/sasl/passwd
postfix reload

The /etc/postfix/sasl/passwd file holds hostname user:pass configurations, one per line:

echo "hostname user:pass" > /etc/postfix/sasl/passwd
chown root:root /etc/postfix/sasl/passwd && chmod 600 /etc/postfix/sasl/passwd

may can be used as a security_level if we are going to send mail to other hosts which may not support security, but make sure that mails are encrypted with talking to the relayhost, maybe through the use of a smtp_tls_policy_maps.

For debugging, you can make SMTP client sessions verbose in Postfix:

smtp      unix  -       -       -       -       -       smtp -v

smtp_sasl_mechanism_filter is also very handy for debugging. For example, you can try to force the authentication mechanism to cram-md5 this way.

Note that this method stores your plaintext password on disk, which is not really desirable. It is best to configure your MUA to send email directly and ignore local emails.

Ideas

  1. ✓ test syncmaildir to replace offlineimap: would allow for an automated configuration based on SSH keys instead of passwords. DONE! this setup is documented in syncmaildir.

  2. ✓ similarly, consider using nullmailer. bremner has this setup that using SSH. nullmailer has the advantage over msmtp that it can queue emails. update: done, with a custom rsendmail, see below.

  3. ✓ alternatively, a simple sendmail wrapper that calls ssh host sendmail could do the job as well. it would need to be some restricted sendmail command, maybe like rsendmail above. but then emails can only be sent online. Update: this was implemented, with a nullmailer remote, as rsendmail.

  4. ultimately, maybe the IMAP server can send the email, through a "Outbox" folder that would slurp emails written there and send them through the SMTP server. this seems to be only supported by Courier IMAP unfortunately.

References

Delivery and retrieval over SSH

I have switched from using IMAP and SMTP to receive and deliver email to SSH as a transport mechanism.

This was originally implemented with syncmaildir (SMD) to replace IMAP, but I have now switched to mbsync. I also wrote rsendmail to replace SMTP.

The my old SMD setup is documented in syncmaildir and the latter is documented in the upstream rsendmail documentation.

This makes it so I do not need to use clear-text passwords to deliver or retrieve email which means everything can be fully automated without writing any password on disk.

Spam filtering

Quick notes on how to configure spam filtering with Spamassassin on Debian.

apt-get install spampd
vi /etc/postfix/master.cf
postfix reload

Modifications required to master.cf:

smtp      inet  n       -       -       -       -       smtpd
        -o content_filter=smtp:127.0.0.1:10025
        -o myhostname=mx.anarcat.ath.cx
localhost:10026        inet    n       -       n       -       10      smtpd
        -o content_filter=
        -o myhostname=delivery.anarcat.ath.cx

In /etc/default/spampd:

ADDOPTS="--tagall --maxsize=1024"

... as the default max size of 64KB is just too small...

In local.cf:

use_bayes 1
bayes_path /var/cache/spampd/bayes
bayes_file_mode 0777
shortcircuit BAYES_99                spam

use_auto_whitelist 1
auto_whitelist_path /var/cache/spampd/awl
# language config, requires Mail::SpamAssassin::Plugin::TextCat
ok_languages de en es fr pt
pyzor_options --homedir /var/cache/spampd

I also load those extra modules in v310.pre:

loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::AWL
loadplugin Mail::SpamAssassin::Plugin::SpamCop
loadplugin Mail::SpamAssassin::Plugin::TextCat

Pyzor needs:

apt install pyzor
sudo -u spampd pyzor --homerdir /var/cache/spampd discover

Bayes autolearning. Spampd cron jobs:

@daily sa-learn --ham --max-size=1048576 /home/anarcat/Maildir/cur/ /home/anarcat/Maildir/.ham/
@daily sa-learn --spam --max-size=0 /home/anarcat/Maildir/.junk/

Fix permissions:

cd Maildir/ &&
chmod -R g+rX . .junk/ .ham/ &&
sudo chown -R :spampd .junk/ .ham/ &&
chmod g+s .junk/* .ham/* &&
chmod g+s tmp &&
sudo chown -R :spampd cur new tmp &&
chmod g+rX cur new tmp -R

First training run:

anarcat@marcos:Maildir$ sudo -u spampd sa-learn --ham --progress --max-size=1048576 ~anarcat/Maildir/cur/
 55% [===============================================================                                                   ]  18.61 msgs/sec 03m03s LEFT

Junk training run:

sudo -u spampd sa-learn --spam --progress --max-size=0 ~anarcat/Maildir/.junk/cur/

To manually check, use:

sudo -u spampd spamassassin -t -d -x <path>

Also, to add to whitelist:

sudo -u spampd spamassassin -t -d -x -W <path>

Also important to enable nightly rules updates:

sudo sed -i s/^CRON=./CRON=1/ /etc/default/spamassassin

This doesn't report emails to pyzor and similar services, unfortunately, see https://wiki.apache.org/spamassassin/ReportingSpam

See also: https://wiki.apache.org/spamassassin/SiteWideBayesFeedback

DMARC / SPF pre-checks

apt install opendmarc

Choose "no" when prompted to configure the database.

The default config is used except for those additions:

--- a/opendmarc.conf
+++ b/opendmarc.conf
@@ -64,7 +64,8 @@ PublicSuffixList /usr/share/publicsuffix/public_suffix_list.dat
 ##  either in the configuration file or on the command line.  If an IP
 ##  address is used, it must be enclosed in square brackets.
 #
-Socket local:/run/opendmarc/opendmarc.sock
+#Socket local:/run/opendmarc/opendmarc.sock
+Socket local:/var/spool/postfix/opendmarc/opendmarc.sock

 ##  Syslog { true | false }
 ##     default "false"
@@ -102,7 +103,7 @@ Syslog true
 ##  specific file mode on creation regardless of the process umask.  See
 ##  umask(2) for more information.
 #
-UMask 0002
+UMask 0007

 ##  UserID user[:group]
 ##     default (none)
@@ -112,3 +113,14 @@ UMask 0002
 ##  the named userid unless an alternate group is specified.
 #
 UserID opendmarc
+
+# ignore submissions through the submission port
+IgnoreAuthenticatedClients true
+# reject stuff like missing From:
+RequiredHeaders true
+# add headers to processed emails
+SoftwareHeader true
+# ignore existing SPF headers that might be spoofed
+SPFIgnoreResults false
+# do the SPF checks as well
+SPFSelfValidate true

Then that socket needs to be added to the postfix configuration:

--- a/postfix/main.cf
+++ b/postfix/main.cf
@@ -119,8 +119,8 @@ compatibility_level = 2
 # OpenDKIM
 milter_protocol = 6
 milter_default_action = accept
-smtpd_milters = local:opendkim/opendkim.sock
-non_smtpd_milters = local:opendkim/opendkim.sock
+smtpd_milters = local:opendkim/opendkim.sock local:opendmarc/opendmarc.sock
+non_smtpd_milters = local:opendkim/opendkim.sock local:opendmarc/opendmarc.sock

Finally, add the postfix user to the opendmarc group so it can access the socket, and restart both servers:

adduser postfix opendmarc
systemctl restart postfix opendmarc

Note that this setup is "soft", in that it will add headers after checking the email but will not outright reject the emails. My idea is to run this in advisory mode for a while and eventually enable rejections, when I'm satisfied this is safe. The magic setting is:

RejectFailures true

Some ideas here are from https://jan.wildeboer.net/2022/09/Email-3-TheRest/.

Todo

To improve on this, I could use the dovecot-antispam plugin, probably by piping messages into sa-learn or some wrapper script. It's unclear how permissions are managed... However, there's a spool2dir method (see upstream manpage) that could be more appropriate, as another daemon could pick up files for training, but it's not in Jessie.

Another thing I could add is the OpenPGP plugin which classifies mail according to its PGP signatures. It fetches keys on the fly and doesn't seem to check for updates. It's also old, so issues may abound.

Finally, we should keep an eye on the rspamd project which reminds me of the old dspam...

Dovecot and mail filters

Postfix is only a MTA, a Mail Transfer Agent. It takes mail from one place and puts it in another place. It doesn't allow you to read mail or do advanced filtering. For this, we need a MDA, a Mail Delivery Agent. The most robust, fastest and powerful server probably available now is Dovecot.

My Dovecot configuration is basically the default, more or less. The key part here is configuring Dovecot to use TLS, otherwise you are sending your password in cleartext all the time. For this you first need to get an X509 certificate somewhere, probably with Let's Encrypt! because it's cheap and automated. You can just reuse the certificate you create through the web server, although you will need special hooks to reload Dovecot when the cert is renewed. I use those simple symlink:

lrwxrwxrwx 1 root root 42 fév  1 19:46 /etc/dovecot/dovecot.pem -> ../letsencrypt/live/anarc.at/fullchain.pem
lrwxrwxrwx 1 root root 43 fév  1 19:42 /etc/dovecot/private/dovecot.pem -> ../../letsencrypt/live/anarc.at/privkey.pem

I also configured filtering and many more things that are documented in 2016-05-12-email-setup.

Todo

On the fly OpenPGP encryption of incoming emails?

Webmail

Yes, people like that thing. Even I like that thing now, because I want to be able to look at my mail without a full IMAP client or logging in through SSH.

I started testing Rainloop, a minimalist webmail client. It does require PHP which sucks, but is way easier to setup than Roundcube and supports mobile very well, while at the same time allowing all the great features you'd expect (sieve, contact lists, search, etc).

First part was to setup PHP. I used PHP-FPM to try to avoid the bloat associated with mod_php. I did this with:

apt install php-fpm
a2enmod proxy_fcgi setenvif
a2enconf php7.0-fpm

Then I created the following config:

<VirtualHost *:80>
    ServerName mail.anarc.at
    ServerAlias imap.anarc.at smtp.anarc.at submission.anarc.at
    Redirect / https://mail.anarc.at/
</VirtualHost>

<VirtualHost *:443>
    ServerName mail.anarc.at
    ServerAlias imap.anarc.at smtp.anarc.at submission.anarc.at
    DocumentRoot /var/www/mail.anarc.at/

    DirectoryIndex /index.php index.php 
    ProxyPassMatch ^/(.*\.php(/.*)?)$ unix:/run/php/php7.0-fpm.sock|fcgi://localhost/var/www/mail.anarc.at

    # protect rainloop configs
    <Directory /var/www/mail.anarc.at/data>
        Options -FollowSymLinks
        AllowOverride None
        <IfVersion >= 2.3>
          Require all denied
        </IfVersion>
        <IfVersion < 2.3>
          Order allow,deny
          Deny from all
        </IfVersion>
    </Directory>
<VirtualHost>

Then I setup the cert with certbot:

certbot certonly --domains mail.anarc.at,imap.anarc.at,smtp.anarc.at,submission.anarc.at --webroot --webroot-path /var/www/mail.anarc.at

... and added the following to the above vhost:

    SSLCertificateFile /etc/letsencrypt/live/mail.anarc.at/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mail.anarc.at/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/mail.anarc.at/chain.pem

And restarted apache of course:

service apache2 reload

Then I setup rainloop, which is disturbingly easy:

wget http://www.rainloop.net/repository/webmail/rainloop-community-latest.zip
mkdir /var/www/rainloop
unzip rainloop-latest.zip -d /var/www/rainloop

Then I visited the admin page (/?admin) and made the following changes:

That must be one of the simplest webapp install I've seen, considering the complexity of this thing. Bravo!

Mailing lists

I naively thought I could go old school and replace Facebook with email (even though I actually never used Facebook). I figured, heck, mailing lists, I know that, I'll just install Mailman 3 in Debian and be done with it.

How wrong can one be. First bug I found was that stretch doesn't have mailman 3, it's only in backports. But then the dependencies in the package are all out of whack (bug #919145, bug #920304). The workaround is simple:

apt install python3-alembic python3-sqlalchemy python3-pymysql python3-mysqldb
apt install -t stretch-backports mailman3-full

Then I found out that the mailman3-web interface is simply uninstallable when using MySQL (reported as bug #921128). So I just used the sqlite3 backend, which is promising to cause delightful problems when interoperating with the mailman3 package, running MySQL.

dpkg-reconfigure mailman3-web

This, incidentally, allows us to have the web server (Apache2) automatically configured, but we won't do that - we'll configure it by hand.

Then Postfix needs to be configured:

owner_request_special = no
transport_maps = hash:/etc/postfix/transport
                 hash:mailman3/postfix_lmtp
local_recipient_maps = proxy:unix:passwd.byname $alias_maps hash:mailman3/postfix_lmtp
relay_domains = ${{$compatibility_level} < {2} ? {$mydestination} : {}} hash:mailman3/postfix_domains

This differs from the configuration suggested in the README because the postfix daemons are usually chrooted (reported as bug #921445). This is then symlinked in place:

touch /var/spool/postfix/mailman3/postfix_domains /var/spool/postfix/mailman3/postfix_lmtp
chown list:list /var/spool/postfix/mailman3/postfix_*
postmap /var/spool/postfix/mailman3/postfix_domains /var/spool/postfix/mailman3/postfix_lmtp
ln -s /var/spool/postfix/mailman3/postfix_domains /var/spool/postfix/mailman3/postfix_lmtp /var/lib/mailman3/data/

And, of course, postfix reloaded:

postfix reload

The data_dir needs to be changed in the mailman config (/etc/mailman3/mailman.cfg), and while you're there change the site_owner as well:

site_owner: anarcat+register@anarc.at
data_dir: /var/spool/postfix/mailman3/

Then we create an Apache config (because the default one kind of sucks):

<VirtualHost *:80>
        ServerName lists.anarc.at
        #Redirect / https://lists.anarc.at/
        DocumentRoot /var/www/html/
</VirtualHost>

<VirtualHost *:443>
        ServerName lists.anarc.at
        #Use common-letsencrypt-ssl lists.anarc.at
        DocumentRoot /var/www/html/
        RedirectMatch ^/$ /mailman3/
        Include /etc/mailman3/apache.conf
</VirtualHost>

Reload apache:

service apache2 reload

A certificate is obtained, after creating the domain of course:

certbot certonly -w /var/www/html -d lists.anarc.at --webroot

Once the cert is enabled, uncomment the Redirect and Use lines and relaod apache again:

service apache2 reload

Finally, a Posterius super user needs to be created:

django-admin createsuperuser --pythonpath /usr/share/mailman3-web --settings settings --username anarcat --email anarcat+register@anarc.at

That will prompt for a password, head over to https://lists.anarc.at/mailman3/ to login. This will ask for an email confirmation, which should confirm your email system somewhat works. Follow that. Then you must configure the Domains to make sure they match the hostname. After you can create a test mailing list and try delivery.

Resolved Issues

I have found out that I had masquerade_domains enabled in my main.cf: bad idea. It made a fool out of myself in bug #921137, where I complained that Mailman would rewrite emails in or out and that would break unsubscribe links and other stuff. I simply disabled that line for now, we'll see what breaks.

Other issues:

Remaining issues

Tested

Future work

Disaster recovery

If the mail server goes down, I currently have one or two fall back options:

  1. migrate email to my shared hosting Koumbit.org account, they support DKIM, SPF, and some DMARC, generally delivers well to Google, some issues with Microsoft (hotmail/office), no spam filtering
  2. use a Riseup email

Other providers that could replace this service include:

new VM setup

This, ideally, should be in Puppet, but in the disaster recover, this shiny puppet server was actually down.

Bootstrap

apt install postfix postfix-pcre 
apt install dovecot-imapd dovecot-sieve dovecot-managesieved
apt install postgrey
apt install opendkim opendkim-tools
apt install opendmarc
apt install chrony certbot

Append this to /etc/postfix/main.cf:

mailbox_command = /usr/lib/dovecot/dovecot-lda -a "$RECIPIENT"
mailbox_size_limit = 0
message_size_limit = 50240000
milter_default_action = accept
milter_protocol = 6
non_smtpd_milters = local:opendkim/opendkim.sock local:opendmarc/opendmarc.sock
smtp_address_preference = ipv4
smtp_dns_support_level = dnssec
smtp_tls_cert_file=${smtpd_tls_cert_file}
smtp_tls_key_file=${smtpd_tls_key_file}
smtp_tls_mandatory_ciphers = high
smtp_tls_security_level = dane
smtpd_helo_required = yes
smtpd_helo_restrictions = reject_non_fqdn_helo_hostname, reject_invalid_helo_hostname
smtpd_milters = local:opendkim/opendkim.sock local:opendmarc/opendmarc.sock
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unknown_client_hostname, reject_non_fqdn_recipient, reject_unauth_destination, reject_unlisted_recipient, check_policy_service inet:127.0.0.1:10023, reject_rbl_client zen.spamhaus.org
smtpd_tls_cert_file = /etc/letsencrypt/live/colette.anarc.at/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/colette.anarc.at/privkey.pem

opendkim:

# write socket inside postfix's chroot
Socket                  local:/var/spool/postfix/opendkim/opendkim.sock

# key/host mappings
SigningTable        refile:/etc/opendkim/signing.table
KeyTable        /etc/opendkim/key.table

# Hosts to generate signatures for
InternalHosts       /etc/opendkim/internal.hosts
# Hosts to ignore when verifying signatures, same as above
ExternalIgnoreList  /etc/opendkim/internal.hosts

Populate:

mkdir -p /var/spool/postfix/opendmarc /etc/opendkim/keys /var/spool/postfix/opendkim
chown opendmarc /var/spool/postfix/opendmarc
chown opendkim /var/spool/postfix/opendkim
adduser postfix opendkim
adduser postfix opendmarc

Config dovecot in conf.d/99-$hostname.conf:

lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
mail_location = maildir:~/Maildir

restore /etc/postfix/master.cf, /etc/postfix/mydestination and /etc/aliases from backups. consider restoring main.cf as well. hell, just restore all of /etc/postfix from backups at this point right? make sure you get /etc/postfix/header_authenticated_redaction

Get a cert:

certbot certonly --standalone -d $(hostname -f)

Enable the cert:

ln -sf /etc/letsencrypt/live/colette.anarc.at/privkey.pem /etc/dovecot/private/dovecot.key
ln -sf /etc/letsencrypt/live/colette.anarc.at/cert.pem /etc/dovecot/private/dovecot.pem

Add the register user:

adduser --disabled-password register

spamd install and config

adduser anarcat spampd
sed -i s/^CRON=./CRON=1/ /etc/default/spamassassin
echo 'ADDOPTS="--maxsize=1024"' >> /etc/default/spampd
cat >> /etc/spamassassin/local.cf

cd Maildir/ &&
chmod -R g+rX . .junk/ .ham/ &&
sudo chown -R :spampd .junk/ .ham/ &&
chmod +gs .junk/* .ham/* &&
chmod +gs tmp &&
sudo chown -R :spampd cur new tmp &&
chmod g+rX cur new tmp -R
Created . Edited .