Read your debian server log and ....

Requirement:

• awk
• ipset & iptables

Read your mail log for authentication failure:
create this bash script ckmail.sh or download 

#!/bin/bash
## create by dedetok April 2016
## GNU GPL v3
echo "reading mail log file"
awk '(/authentication failure/) { print $7," ",$(NF-1)," ",$(NF)}' /var/log/mail
.log | sort | uniq -c | {
  while read -r line1 line2 line3 line4
  do
    if [ "$line1" -ge 5 ]; then
       echo -e "$line1\t$line2 $line3 $line4"
    fi
  done
}

create ipset rules for blacklisting smpt

# ipset create mynetrulessmtpd hash:net

Run script and pupulate mynetrulessmtpd:

# ./ckmail.sh
reading mail log file
20      unknown[157.122.148.154]: authentication failure
840     unknown[195.22.127.187]: authentication failure
16      unknown[91.193.74.31]: authentication failure
# ipset add mynetrulessmtpd 157.122.148.154
# ipset add mynetrulessmtpd 195.22.127.187
# ipset add mynetrulessmtpd 91.193.74.31

Add ipset rules into iptables rules

# iptables -I INPUT -p tcp --match multiport --dports smtp,smtps -m set --match-set mynetrulessmtpd src -j DROP