Loading ipset and iptables on boot in Debian Jessie

Debian Jessie until I write this note, does not provide script to save and load ipset rules.
ipset
-----------------------------------------------------------------------------------------------
saving ipset rules:

# ipset save 

or

# /sbin/ipset save

We follow wiki rules for filename https://wiki.debian.org/iptables (i.e. /etc/iptables.up.rules)

# ipset save > /etc/ipset.up.rules

To save ipset rules to other file

# ipset save > /root/iptablesrules/ipsetrules.save 

To restore ipset rules

# ipset restore -! < /etc/ipset.up.rules

or

# /sbin/ipset restore -! < /etc/ipset.up.rules

iptables (my version v1.4.21)
-----------------------------------------------------------------------------------------------
saving iptables rules

# iptables-save 

or

# /sbin/iptables-save 

We follow wiki rules for filename https://wiki.debian.org/iptables (i.e. /etc/iptables.up.rules)

# iptables-save > /etc/iptables.up.rules

To save iptables rules to other file

# iptables-save > /root/iptablesrules/iptablesrules.save

Note: in iptables-persistent packages, rules are save in file:

1. /etc/iptables/rules.v4
2. /etc/iptables/rules.v6

But we do not use iptables-persistent
To restore iptables

# iptables-restore < /etc/iptables.up.rules

or

# /sbin/iptables-restore < /etc/iptables.up.rules

We have some choice to load ipset and iptables on boot:

1. Manual init.d configuration
2. Configuring via ifup
1. put loader in file /etc/network/interface
2. put script configuration in /etc/network/if-pre-up.d/

Option 1:

To use /etc/network/interface to load ipset and iptables
Edit /etc/network/interface

....
# The primary network interface
allow-hotplug eth0
iface eth0 inet static
   ...
   pre-up ipset restore -! < /etc/ipset.up.rules
   pre-up iptables-restore < /etc/iptables.up.rules
...

Option 2:

To use script configuration in /etc/network/if-pre-up.d/ to load ipset and iptables
Create or edit /etc/network/if-pre-up.d/load.rules

#!/bin/sh
/sbin/ipset restore -! < /etc/ipset.up.rules
/sbin/iptables-restore < /etc/iptables.up.rules

chmod +x /etc/network/if-pre-up.d/load.rules

Note:

1. Use only one option above.
2. After adding ip address into ipset rules, don't forget to save it in file etc/ipset.up.rules.
3. If you use fail2ban, Do not put fail2ban rules in iptables.up.rules. It will automatically configure it self. you need to remove fail2ban rules in file /etc/iptables.up.rules.

Distributing ipset across server farm
This ipset rules can be distributed across your server.

Master ipset can be generated from honeypot/server and distribute it via web. Run this script after you add or edit ipset rules into your root web directory

#!/bin/bash

## create by dedetok April 2016

## last update 2016-04-28

## GNU GPL v3

## Disclaimer: experimental, use it with your own risk

/sbin/ipset save > /etc/ipset.up.rules

# create temporary file to save new ipset rules without fail2ban rules

if [ -f "/root/bin/ipset.up.rules.new" ] ; then

 rm "/root/bin/ipset.up.rules.new"

fi

touch /root/bin/ipset.up.rules.new

while read -r line; do

 if [[ $line != *"fail2ban"* ]]

 then

  echo "$line" >> /root/bin/ipset.up.rules.new

 fi

done < /etc/ipset.up.rules

# copy clean ipset into /etc/ipset.up.rules

cp /root/bin/ipset.up.rules.new /etc/ipset.up.rules

# save it into web or user public_html

#cp /root/bin/ipset.up.rules.new /home/[user]/public_html/ipset.up.rules

cp /root/bin/ipset.up.rules.new /var/www/public_html/ipset.up.rules

Do the following steps on your server farm:

• Write this bash script to download /root/unduhgarasiku.sh or download

#!/bin/bash
## create by dedetok April 2016
## last update 2017-05-02
## GNU GPL v3
## Disclaimer: experimental, use it with your own risk
echo "download from http://www.garasiku.web.id/ipset.up.rules into /etc/ipset.up.rules.new"
if wget -O /etc/ipset.up.rules.new http://www.garasiku.web.id/ipset.up.rules; then
 chmod 444 /etc/ipset.up.rules.new
 chown root:root /etc/ipset.up.rules.new
 ## Update ipset ignore error, we need fresh list
 echo "updating new rules"
 if /sbin/ipset restore -! < /etc/ipset.up.rules.new; then
  echo "Saving new ipset rules into /etc/ipset.up.rules"
  cp /etc/ipset.up.rules.new /etc/ipset.up.rules
  chmod 544 /etc/ipset.up.rules
  chown root:root /etc/ipset.up.rules
 else
  echo "Error, ipset.up.rules not in ipset format"
  exit 1
 fi
else
 echo "Fail to download ipset.up.rules"
 exit 1
fi
echo "End process"

Old version

#!/bin/bash
## create by dedetok April 2016
## last update 2016-04-15
## GNU GPL v3
## Disclaimer: experimental, use it with your own risk 
echo "download from http://www.garasiku.web.id/ipset.up.rules into /etc/ipset.up.rules.new"
wget -O /etc/ipset.up.rules.new http://www.garasiku.web.id/ipset.up.rules
chmod 444 /etc/ipset.up.rules.new
chown root:root /etc/ipset.up.rules.new
## Compare ipset.up.rules vs ipset.uprules.new
echo "updating new rules"
diff --new-line-format="+ %L" --old-line-format="- $L" <(sort /etc/ipset.up.rules) <(sort /etc/ipset.up.rules.new) |
while IFS=' ' read -r r1 r2 r3 r4; do
 if [ "$r2" = "add" ]; then
  if [ "$r1" = "+" ]; then
   cmdline="/sbin/ipset $r2 $r3 $r4"
   echo "eval $cmdline"
   eval "$cmdline"
  fi
  if [ "$r2" = "-" ]; then
   cmdline="/sbin/ipset del $r3 $r4"
   echo "eval $cmdline"
   eval "$cmdline"
  fi
 fi
done
echo "Saving new ipset rules into /etc/ipset.up.rules"
/sbin/ipset save > /etc/ipset.up.rules
echo "End process"

• Put it into crontab to update ipset.up.rules everyday at 0 night:

# crontab -e

• Put this line (you can choose nano editor)

0 0 * * * /root/unduhgarasiku.sh

• Save it (you can use default file name)

File Name to Write: /tmp/crontab.9uLsb5/crontab

• iptabes rules for server farm

-A INPUT -m set --match-set mynetrules src -j DROP

-A INPUT -p tcp -m multiport --dports 25,465,587,993,995,465,143,110 -m set --match-set mynetrulessmtp src -j DROP

-A INPUT -p tcp -m multiport --dports 80,443 -m set --match-set mynetruleshttp src -j DROP

-A INPUT -p tcp --dport 22 -m set --match-set mynetrulesssh src -j DROP

-A INPUT -p tcp  -m multiport --dports 21,22 -m set --math-set mynetrulesftp src -j DROP
-A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 3 -j DROP
or if you want to limiting connection from class C up to 20 connection -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 20 --connlimit-mask 24 -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m multiport --dports 25,465,587,993,995,265,143,110 -m connlimit --connlimit-above 8 --connlimit-mask 32 --connlimit-saddr -j DROP

NOTE: don't forget to add your ipset rules into iptables rules in every your server farms and make it persistent. I suggest you create iptables rules in every server, do not copy iptables other server. Every server may have unique iptables rules.

Simple script to analyst authentication log files:

• Search fail in sshawk '(/authentication fail/ && /ssh/) { print $(NF-1), $NF }' /var/log/auth.log | sort | uniq -c | sort -n

• search fail in smtpawk '(/authentication fail/ && /smtp/) { print $(NF-1), $NF }' /var/log/auth.log | sort | uniq -c | sort -n

• search fail in ftpawk '(/authentication fail/ && /ftp/) { print $(NF-1), $NF }' /var/log/auth.log | sort | uniq -c | sort -n

• search fail in dovecotawk '(/authentication fail/ && /dovecot/) { print $(NF-1), $NF }' /var/log/auth.log | sort | uniq -c | sort -n

• search ssh preauthentication awk '(/Connection closed by/ && /sshd/) { print $(NF-1),$NF}' /var/log/auth.log | sort | uniq -c | sort -n
  awk '(/preaut/ && /sshd/) { print $0}' /var/log/auth.log
• search fail in mailawk '(/authentication fail/) { print $7}' /var/log/mail.log | sort | uniq -c

Reference:

• http://www.debian.org/doc/manuals/securing-debian-howto/ch-sec-services.en.html#s-firewall-setup
• http://bencane.com/2011/12/30/when-its-ok-and-not-ok-to-use-rc-local/
• http://superuser.com/questions/802253/most-secure-way-to-have-iptables-auto-loaded-using-debian-linux
• http://forums.debian.net/viewtopic.php?f=5&t=111917
• https://major.io/2009/11/16/automatically-loading-iptables-on-debianubuntu/
• http://www.cyberciti.biz/faq/iptables-connection-limits-howto/
• https://wiki.debian.org/iptables

How to determine your Debian need to restart after packages update

Does your Debian server need to restart after packages update?

You can use checkrestart from the debian-goodies. debian-goodies availabe since Debian Wheezy.

• Install debian-goodies

# apt-get install debian-goodies

• Check does our Debian need to restart

# checkrestart
Found 44 processes using old versions of upgraded files
(28 distinct programs)
(21 distinct packages)
...

• Restart your Debian

References:

• http://serverfault.com/questions/667076/debian-how-can-i-know-if-reboot-is-required-after-update

Find host or IP on Fail2ban log in Centos using bash and awk

Find out your fail2ban log and make permanent block. You need to install ipset to make your iptables rules

---

• Create script bannedge2centos.sh

#!/bin/bash
## create by dedetok March 2016
## GNU GPL v3
echo "These IP get WARNING at least twice"
awk '(/fail2ban.filter/ && /WARNING/) { print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort -n |
{
  while read -r line1 line2
  do
    if [ "$line1" -ge 2 ]; then
      echo "$line1 $line2"
    fi
  done
}

echo "These IP get Filter sshd at least twice"
awk '(/fail2ban.filter/ && /sshd/) { print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort -n | {
  while read -r line1 line2
  do
    if [ "$line1" -ge 2 ]; then
      echo "$line1 $line2"
    fi
  done
}

echo "These IP get Filter ftpd at least twice"
awk '(/fail2ban.filter/ && /vsftpd/) { print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort -n | {
  while read -r line1 line2
  do
    if [ "$line1" -ge 2 ]; then
      echo "$line1 $line2"
    fi
  done
}

echo "These hosts get Filter Fail2ban at least twice"
awk '(/fail2ban.filter/ && /WARNING Unable/) { print $(NF-7)}' /var/log/fail2ban.log | sort | uniq -c |
 sort -n | {
  while read -r line1 line2
  do
    if [ "$line1" -ge 2 ]; then
      echo "$line1 $line2"
    fi
  done
}

• Create ipset rules mynetrules

# ipset create mynetrules hash:net

• Run bash bannedge2centos.sh  

# ./bannedge2centos.sh
These IP get WARNING at least twice
2 ['199.15.112.8']
2 ['216.117.2.180']
2 ['82.148.206.193']
4 ['208.100.26.231']
4 ['31.211.102.129']
5 ['71.6.167.142']
10 ['88.80.15.69']
28 known
These IP get Filter sshd at least twice
2 118.136.248.90
18 84.16.74.40
These IP get Filter ftpd at least twice
5 88.80.15.69
These hosts get Filter Fail2ban at least twice
2 128-140-19-52.maxnet.ir:
2 1-39-45-166.live.vodafone.in:
2 178.218.202.224.ip.turontelecom.uz:
2 181-174-60-157.telebucaramanga.net.co:
2 222.64.uzpak.uz:
2 238.124.206.49-ras.beamtele.net:
2 75-242-69-115.vasaicable.co.in:
2 78.187.230.16.static.ttnet.com.tr:
2 85-113-26-150.static.ktnet.kg:
2 91-219-55-106.static-pool.centr.zp.ua:
2 brbnd47-30.mng.net:
2 static-mum-120.63.188.93.mtnl.net.in:
4 client.fttb.2day.kz:

• Popuate ipset rules mynetrules

# ipset add mynetrules 84.16.74.40
# ipset add mynetrules 71.6.167.142
# ipset add mynetrules 208.100.26.231
# ipset add mynetrules 31.211.102.129

• Add ipset rules into iptables rules

# iptables -I INPUT -m set --match-set mynetrules src -j DROP

• Save your new rules

# service ipset save
# service iptables save
Default ipset and iptables: /etc/sysconfig/ipset, /etc/sysconfig/iptables,  and /etc/sysconfig/iptables.save 

• To reload ipset and iptables

# service ipset reload
# service iptables restart

Another interesting scrips

• # awk '(/pam_unix/ && /sshd:auth/ && /authentication failure/) { print $NF }' /var/log/secure | sort | uniq -c | sort
• # awk '(/pam_unix/ && /webmin/ && /authentication failure/) { print $(NF-1) }' /var/log/secure | sort | uniq -c | sort
• # awk '(/pam_unix/ && /vsftpd/ && /authentication failure/) { print $(NF) }' /var/log/secure | sort | uniq -c | sort 
• awk '((/not receive/ || /Failed/ || /Connection closed/ || /closed/ || /failure/ ) && /preauth/ && /sshd/) { print (NF-2),$(NF-1),$NF}' /var/log/auth.log | sort  | uniq -c

Note:

• You can modify these to auto block those IP into your ipset rules. But beware about your IP may incidentally get auto blocked. Check those IP or host location before add them into ipset ruls
• When ipset size increasing, it wont impose to CPU load.

Read your debian server log and ....

Requirement:

• awk
• ipset & iptables

Read your mail log for authentication failure:
create this bash script ckmail.sh or download 

#!/bin/bash
## create by dedetok April 2016
## GNU GPL v3
echo "reading mail log file"
awk '(/authentication failure/) { print $7," ",$(NF-1)," ",$(NF)}' /var/log/mail
.log | sort | uniq -c | {
  while read -r line1 line2 line3 line4
  do
    if [ "$line1" -ge 5 ]; then
       echo -e "$line1\t$line2 $line3 $line4"
    fi
  done
}

create ipset rules for blacklisting smpt

# ipset create mynetrulessmtpd hash:net

Run script and pupulate mynetrulessmtpd:

# ./ckmail.sh
reading mail log file
20      unknown[157.122.148.154]: authentication failure
840     unknown[195.22.127.187]: authentication failure
16      unknown[91.193.74.31]: authentication failure
# ipset add mynetrulessmtpd 157.122.148.154
# ipset add mynetrulessmtpd 195.22.127.187
# ipset add mynetrulessmtpd 91.193.74.31

Add ipset rules into iptables rules

# iptables -I INPUT -p tcp --match multiport --dports smtp,smtps -m set --match-set mynetrulessmtpd src -j DROP

ipset: iptables extensions

ipset is a framework that introduces since kernel 2.4.

Installation
To install it in debian jessie

# apt-get install ipset

Creating rules
To create a new set of ipset using TYPENAME hash:net (The hash:net set type uses a hash to store different sized IP network addresses. Network address with zero prefix size cannot be stored in this type of sets.)

# ipset create mynetrules hash:net

or if you want to create set of ipset using TYPENAME hash:ip (The hash:ip set type uses a hash to store IP host addresses (default) or network addresses. Zero valued IP address cannot be stored in a hash:ip type of set.)

# ipset create myiprules hash:ip

or you want to create set of ipset using TYPENAME hash:ip,port (The hash:ip,port set type uses a hash to store IP address and port number pairs. The port number is interpreted together with a protocol (default TCP) and zero protocol number cannot be used.)

# ipset create myipportrules hash:ip,port

"If you want to store same size subnets from a given network (say /24 blocks from a /8 network), use the bitmap:ip set type. If you want to store random same size networks (say random /24 blocks), use the hash:ip set type. If you have got random size of netblocks, use hash:net."

View available ipset rules

To view available ipset rules 

# ipset list

Adding IP into ipset rules

To add single IP into ipset rules:

# ipset add myiprules 183.3.202.105

To add block IP into ipset rules:

# ipset add myiprules 193.201.227.0/24

If host names or service names with dash in the name are used instead of IP addresses or service numbers, then the host name or service name must be enclosed in square brackets. Example:

# ipset add myiprules [u18576666.onlinehome-server.com]

Removing IP from ipset rules
To remove ip or hostname from ipset rules

# ipset del myiprules [u18576666.onlinehome-server.com]

or

# ipset del myiprules 183.3.202.105

or

# ipset del myiprules 193.201.227.0/24

Adding ipset rules into iptables
To add ipset rules into iptables and drop it

# iptables -I INPUT -m set --match-set mynetrules src -j DROP

or if you want just to block connection to ssh 

# iptables -I INPUT  -p tcp --dport 22 -m set --match-set mynetrules src -j DROP

Note: 

-I to insert rules into first line

-A to insert rules into last line

To delete rules, replace -I or -A to -D

How to restore or refresh ipset rules if rule name exist 

To show list of iptables 

# ipset -exist restore < /root/ipset.up.rules

Additional command

To show list of iptables 

# iptables -L

To show list of iptables with line number

# iptables -L --line-numbers

To show list INPUT of iptables 

# iptables -L INPUT

To show list INPUT of iptables with line number

# iptables -L INPUT --line-numbers

To delete line 2 from INPUT

# iptables -D INPUT 2

 

Help

# ipset --help
ipset v6.23

Usage: ipset [options] COMMAND

Commands:
create SETNAME TYPENAME [type-specific-options]
        Create a new set
add SETNAME ENTRY
        Add entry to the named set
del SETNAME ENTRY
        Delete entry from the named set
test SETNAME ENTRY
        Test entry in the named set
destroy [SETNAME]
        Destroy a named set or all sets
list [SETNAME]
        List the entries of a named set or all sets
save [SETNAME]
        Save the named set or all sets to stdout
restore
        Restore a saved state
flush [SETNAME]
        Flush a named set or all sets
rename FROM-SETNAME TO-SETNAME
        Rename two sets
swap FROM-SETNAME TO-SETNAME
        Swap the contect of two existing sets
help [TYPENAME]
        Print help, and settype specific help
version
        Print version information
quit
        Quit interactive mode

Options:
-o plain|save|xml
       Specify output mode for listing sets.
       Default value for "list" command is mode "plain"
       and for "save" command is mode "save".
-s
        Print elements sorted (if supported by the set type).
-q
        Suppress any notice or warning message.
-r
        Try to resolve IP addresses in the output (slow!)
-!
        Ignore errors when creating or adding sets or
        elements that do exist or when deleting elements
        that don't exist.
-n
        When listing, just list setnames from the kernel.

-t
        When listing, list setnames and set headers
        from kernel only.
-f
        Read from the given file instead of standard
        input (restore) or write to given file instead
        of standard output (list/save).

Supported set types:
    list:set            3       skbinfo support
    list:set            2       comment support
    list:set            1       counters support
    list:set            0       Initial revision
    hash:mac            0       Initial revision
    hash:net,iface      6       skbinfo support
    hash:net,iface      5       forceadd support
    hash:net,iface      4       comment support
    hash:net,iface      3       counters support
    hash:net,iface      2       /0 network support
    hash:net,iface      1       nomatch flag support
    hash:net,iface      0       Initial revision
    hash:net,port       7       skbinfo support
    hash:net,port       6       forceadd support
    hash:net,port       5       comment support
    hash:net,port       4       counters support
    hash:net,port       3       nomatch flag support
    hash:net,port       2       Add/del range support
    hash:net,port       1       SCTP and UDPLITE support
    hash:net,port,net   2       skbinfo support
    hash:net,port,net   1       forceadd support
    hash:net,port,net   0       initial revision
    hash:net,net        2       skbinfo support
    hash:net,net        1       forceadd support
    hash:net,net        0       initial revision
    hash:net            6       skbinfo support
    hash:net            5       forceadd support
    hash:net            4       comment support
    hash:net            3       counters support
    hash:net            2       nomatch flag support
    hash:net            1       Add/del range support
    hash:net            0       Initial revision
    hash:ip,port,net    7       skbinfo support
    hash:ip,port,net    6       forceadd support
    hash:ip,port,net    5       comment support
    hash:ip,port,net    4       counters support
    hash:ip,port,net    3       nomatch flag support
    hash:ip,port,net    2       Add/del range support
    hash:ip,port,net    1       SCTP and UDPLITE support
    hash:ip,port,ip     5       skbinfo support
    hash:ip,port,ip     4       forceadd support
    hash:ip,port,ip     3       comment support
    hash:ip,port,ip     2       counters support
    hash:ip,port,ip     1       SCTP and UDPLITE support
    hash:ip,mark        2       sbkinfo support
    hash:ip,mark        1       forceadd support
    hash:ip,mark        0       initial revision
    hash:ip,port        5       skbinfo support
    hash:ip,port        4       forceadd support
    hash:ip,port        3       comment support
    hash:ip,port        2       counters support
    hash:ip,port        1       SCTP and UDPLITE support
    hash:ip             4       skbinfo support
    hash:ip             3       forceadd support
    hash:ip             2       comment support
    hash:ip             1       counters support
    hash:ip             0       Initial revision
    bitmap:port         3       skbinfo support
    bitmap:port         2       comment support
    bitmap:port         1       counters support
    bitmap:port         0       Initial revision
    bitmap:ip,mac       3       skbinfo support
    bitmap:ip,mac       2       comment support
    bitmap:ip,mac       1       counters support
    bitmap:ip,mac       0       Initial revision
    bitmap:ip           3       skbinfo support
    bitmap:ip           2       comment support
    bitmap:ip           1       counters support
    bitmap:ip           0       Initial revision

 

References:

• http://daemonkeeper.net/781/mass-blocking-ip-addresses-with-ipset/ 
• http://nemgeek.blogspot.co.id/2013/07/ipset-for-heavy-use.html 
• https://mattwilcox.net/web-development/unexpected-ddos-blocking-china-with-ipset-and-iptables 
• http://www.linuxjournal.com/content/advanced-firewall-configurations-ipset 
• http://ipset.netfilter.org/ 
• http://ipset.netfilter.org/ipset.man.html