I hate email spam and not just receiving but sending it, especially if it involves my own mail server. I’ve noticed in the recent weeks, that the amount of mail log produced on my mail server is at an all time high. Upon investigation it showed these are the result of unsuccessful login attempts. So I created a bunch of scripts to automatically discard those.

Looking at the logs there were dozen attempts a minute to connect to my server from various locations. Even if these attempts failed it is not a good idea to have these around. These attempts eat up bandwidth and CPU times as well as expose the server to a brute force password based attack.

I could blacklist these senders one-by-one manually, but that assumes I want to keep up to date with all the current spam sender botnets. I’d chosen a different way, going with the idea of blacklisting these senders automatically from time to time.

Looking at the possibilities of existing solutions I found none to fit my needs.

My server is running a vanilla Ubuntu server with a Postfix mailserver and a ufw firewall set up. The syslog has no special settings, and logrotate is as it is coming from the box. The guide assumes you have a similar setup and you know what you are doing. (NB. This is a quite standard, so my approach might be useful to others, but be aware, YMMV and I take no responsibility if it doesn’t work for you.)

First I decided to take the approach of collecting all attempts that only had unsuccessful authentication tries from the mailserver’s log. Than using a simple awk script I count the frequency of connection attempts and discard those under a threshold. I do this for both ip4 and ip6 as spammers recently started using the later.

The second part of the exercise is adding a few lines of code to the ufw startup scripts, that will later be used for blocking invalid connection attempts altogether. These lines will create a harness on the mail server ports and add a sink rule section to log the denied connections.

In the /etc/ufw/before.rules, just after the line

# End required lines

add the two new rule groups

:ufw-smtp-blacklist - [0:0]
:ufw-smtp-blacklist-log - [0:0]

scroll down to the end of the file and add the following block to define the actual rules before the commit line.

-A ufw-before-input -p tcp --dport 25 -j ufw-smtp-blacklist
-A ufw-before-input -p tcp --dport 465 -j ufw-smtp-blacklist
-A ufw-before-input -p tcp --dport 587 -j ufw-smtp-blacklist

## BLACKLIST-START ##
## BLACKLIST-END ##

-A ufw-smtp-blacklist-log -j LOG --log-prefix "[SMTP BLACKLIST] " -m limit --limit 3/min --limit-burst 10
-A ufw-smtp-blacklist-log -j DROP

For supporting the IP6 do the same for /etc/ufw/before6.rules just after the line

# End required lines

add the lines

:ufw6-smtp-blacklist - [0:0]
:ufw6-smtp-blacklist-log - [0:0]

And add the following before the commit line.

-A ufw6-before-input -p tcp --dport 25 -j ufw6-smtp-blacklist
-A ufw6-before-input -p tcp --dport 465 -j ufw6-smtp-blacklist
-A ufw6-before-input -p tcp --dport 587 -j ufw6-smtp-blacklist

## BLACKLIST-START ##
## BLACKLIST-END ##

-A ufw6-smtp-blacklist-log -j LOG --log-prefix "[SMTP BLACKLIST] " -m limit --limit 3/min --limit-burst 10
-A ufw6-smtp-blacklist-log -j DROP

Then we create a little generic helper script to count the occurance of each line in an input. For now place it in /root/frequency.awk (however I advise you to run it as another user and use sudo for execution, that part I will leave you for your imagination)

#!/usr/bin/gawk -f
{
        freq[$0]++
}

END {
        for (word in freq)
                printf "%s\t%d\n", word, freq[word]
}

To be able to blackhole entire subdomains I also created a little helper that concatenates the frequencies to domains. Place it under your runtime folder, and name it ip_group.awk,

#!/usr/bin/gawk -f
# assume input ip feequency
# group by ip domain
{
        t = $1;
        sub(/\.[0-9]+$/, "", t)
        x[t]+=$2
        x2[t]++
        y[$1]=$2
}
END {
        for (i in y) {
                t=i
                sub(/\.[0-9]+$/, "", t)
                if (x2[t]<5)
                        print i, y[i]
        }
        for (i in x) {
                if (x2[i]>=5)
                        printf ("%s.0/24 %s\n", i, x[i])
        }
}

Postfix rejections

I create a separate config for postfix for the more relaxed and humane reject option. The idea was to reject connection attempts from these domains first, before forcibly cutting them off using the firewall. However I found it is not really worth notifying these spammers about rejection. I left the option there, so if you are interested you could introduce two levels of rejections.

To make postfix rejection work, create a file named /etc/postfix/sender access with the content

## GENERATED ##
## GENERATED ##

You can also add other permit or reject rules in this file.

When that’s done add the following restrictions to your main.cf as an early rule.

smtpd_client_restrictions =
  check_client_access hash:/etc/postfix/sender_access
  .. 
smtpd_recepient_restrictions =
  check_client_access hash:/etc/postfix/sender_access
  ..

Blocking rules

What’s left is to create the actual blocking rules based on the attempts. Put it in your favorite folder and make sure it is executable. For the sake of demonstration I assume it is in the root folder and is called sender_blacklist.sh.

In the script we first scan the log for failed authentication attempts with ip4 addresses. Using the frequency counter script we only take those over a certain threshold. Since logrotate will reset these counts every day, you will not permanently blacklist any domain permanently.

The same is done for ip6 addresses, with a much more aggressive threshold as that is currently not used normally on my server.

The resulting list is used to generate and replace the postfix and the ufw blacklists in their respective files and restart the services.

#!/bin/bash
DEBUG=1
RETRIES4=10
RETRIES6=1
LIST4="$(grep 'SASL LOGIN authentication failed' /var/log/mail.warn|
        grep -E '\[[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\]'|
        sed -r 's/.* (.*)\[([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\].*/\1 \2/'|
        /root/linefreq.awk|
        sort|
        awk '{if ($3>'$RETRIES4') print $0}')"
LIST6="$(grep 'SASL LOGIN authentication failed' /var/log/mail.warn|
        grep -E '\[[0-9a-f]+\:([0-9a-f]+\:)+[0-9a-f]+\]'|
        sed -r 's/.* (.*)\[([0-9a-f]+\:([0-9a-f]+\:)+[0-9a-f]+)\].*/\1 \2/'|
        /root/linefreq.awk|
        sort|
        awk '{if ($3>'$RETRIES6') print $0}')"

echo To blacklist
echo "$LIST4"
echo "$LIST6"
POSTFIX_BLACKLIST="$(echo -e "$LIST4\n$LIST6"|
        gawk 'BEGIN{printf "## GENERATED ##\\n"}{if ($2) {printf ("%s REJECT Too many connection attempts %s from %s(%s)\\n", $2,$3,$1,$2)}} END{printf "## GENERATED ##"}')"

if [ ! -z "$LIST4" ]; then
        IPTABLES_BLACKLIST4="$(echo "$LIST4"|
                gawk '{printf ("-A ufw-smtp-blacklist --src %s -j ufw-smtp-blacklist-log\\n", $2)}')"
fi

if [ ! -z "$LIST6" ]; then
        IPTABLES_BLACKLIST6="$(echo "$LIST6"|
                gawk '{printf ("-A ufw6-smtp-blacklist --src %s -j ufw6-smtp-blacklist-log\\n", $2)}')"
fi

if [[ "$DEBUG" != 0 ]]; then
        echo $POSTFIX_BLACKLIST
        echo $IPTABLES_BLACKLIST4
        echo $IPTABLES_BLACKLIST6
        exit
fi

IPTABLES_BLACKLIST4="## BLACKLIST-START ##\n$IPTABLES_BLACKLIST4## BLACKLIST-END ##"
IPTABLES_BLACKLIST6="## BLACKLIST-START ##\n$IPTABLES_BLACKLIST6## BLACKLIST-END ##"

sed -i '/## GENERATED ##/,/## GENERATED ##/c\'"${POSTFIX_BLACKLIST}"  /etc/postfix/sender_access
sed -i '/## BLACKLIST-START ##/,/## BLACKLIST-END ##/c\'"${IPTABLES_BLACKLIST4}" /etc/ufw/before.rules >/dev/null
sed -i '/## BLACKLIST-START ##/,/## BLACKLIST-END ##/c\'"${IPTABLES_BLACKLIST6}" /etc/ufw/before6.rules >/dev/null

/usr/sbin/postmap /etc/postfix/sender_access
/usr/sbin/postfix reload
/usr/sbin/ufw reload

To test how it would work on your system, you can use the DEBUG environment variable. It is deliberately set to 1 in the script above.

# /sender_blacklist.sh

This will list all the hosts and their respective names it would block so you can simulate what would it do. It also dumps the contents it would put in the files without actually performing any changes.

Depending on your system this may be a huge list! On my system processing the log takes around 1.2s.

The script is ran in memory, so do not attempt to use it on heavy loaded machines, but modify to use temporary files instead.

Once it seems to work well for your system, you can set DEBUG to 0 and see if you broke something.

What really remains is testing your rules on your machine. If all works well several hosts will instantly get blacklisted and your logs will contain rejections.

To see what got rejected a simple statistics can be pulled with the following scriptlet

#!/bin/bash
grep BLACKLIST /var/log/syslog | \
        sed -r 's/.*SRC=(.*) DST.*/\1/' | \
        /root/linefreq.awk|\
        sort -nr -k2

Once you are happy with the results you can add the script to your crontab. I run it every fifteen minutes to build up the blacklists. It won’t prevent every spammer but reduces the load on the server.