Core util: cat, head, tail, sort, uniq and cut

cat

cat copies each file (‘ -’ means standard input), or standard input if none are given, to standard output.
Synopsis:

cat [ option ] [ file]…

head

head prints the first part (10 lines by default) of each file; it reads from standard input if no files are given or when given a file of - .
Synopsis:

head [ option ]… [ file]…

# head /var/log/auth.log

tail

tail prints the last part (10 lines by default) of each file; it reads from standard input if no files are given or when given a file of ‘ -’.
Synopsis:

tail [ option ]… [ file]…

# tail /var/log/auth.log

sort

sort sorts, merges, or compares all the lines from the given files, or standard input if none are given or for a file of ‘ - ’. By default, sort writes the results to standard output.
Synopsis:

sort [ option ]… [ file]…

options:

‘ -n ’
‘ --numeric-sort ’
‘ --sort=numeric ’
Sort numerically. The number begins each line and consists of optional blanks, an optional ‘ -’ sign, and zero or more digits possibly separated by thousands separators, optionally followed by a decimal-point character and zero or more digits. An empty number is treated as ‘ 0 ’. The LC_NUMERIC locale specifies the decimal-point character and thousands separator. By default a blank is a space or a tab, but the LC_CTYPE locale can change this.

uniq

uniq writes the unique lines in the given
input , or standard input if nothing is given or for an input name of ‘ -’.
Synopsis:

uniq [ option ]… [ input [output ]]

options:

‘ -c ’
‘ --count ’
Print the number of times each line occurred along with the line.

cut

cut writes to standard output selected parts of each line of each input file, or standard input if no files are given or for a file name of ‘ - ’.
Synopsis:

cut option… [ file]…

Options:
‘ -d input_delim_byte ’
‘ --delimiter= input_delim_byte ’
With -f, use the first byte of
input_delim_byte as the input fields separator (default is TAB).

‘ -f field-list ’

‘ --fields= field-list ’

Select for printing only the fields listed in field-list . Fields are separated by a TAB character by default. Also print any line that contains no delimiter character, unless the --only-delimited (-s) option is specified.

Note awk supports more sophisticated field processing, and by default will use (and discard) runs of blank characters to separate fields, and ignore leading and trailing blanks.

awk '{print $2}'    # print the second field
awk '{print $NF-1}' # print the penultimate field
awk '{print $2,$1}' # reorder the first two fields

In the unlikely event that awk is unavailable, one can use the join command, to process blank characters as

awk does above.

join -a1 -o 1.2     - /dev/null # print the second field
join -a1 -o 1.2,1.1 - /dev/null # reorder the first two fields

Example: a quick way to see which IP addresses are most active is to sort by them:

# cat access.log |cut -d ' ' -f 1 |sort

UPDATE: even easier: the uniq command has a -c argument that does most of this work automatically. It counts the occurrences of each unique line. Then a quick sort -n and a tail shows the big ones. Also, I tend to use "cut" as above, but one of the Dreamhost guys reminded me that awk may be a little more straightforward:

# cat /path/to/access.log |awk '{print $1}' |sort  |uniq -c |sort -n |tail

References:

• https://www.gnu.org/software/coreutils/manual/html_node/index.html
• https://encodable.com/tech/blog/2008/12/17/Count_IP_Addresses_in_Access_Log_File_BASH_OneLiner

Virtualmin creating Sub-Server for Sub-Domain

These are steps to create Sub-Server for Sub-Domain:

1. Virtualmin -> Create Virtual Server
2. Fill Domain name for sub-domain for example subdom.domain.com (domain.com is root domain, change with your root domain) 
3. In Enabled features check these items:
   1. Setup DNS zone?
   2. Setup website for domain?
   3. Setup SSL website too?
   4. Choose one of these Setup Webalizer for web logs? or Enable AWstats reporting? (optional)
   5. Choose other options depend on your requirement

Linux awk

Awk can do most things that are actually text processing.

An awk program follows the form:

pattern { action }

awk is line oriented. That is, the pattern specifies a test that is performed with each line read as input. If the condition is true, then the action is taken. The default pattern is something that matches every line. This is the blank or null pattern.
awk program below:

BEGIN { print "START" }       { print         } END   { print "STOP"  }

Example:

BEGIN { print "File\tOwner"}{ print $8, "\t", $3}END { print " - DONE -" }

Example awk_example1.awk

#!/bin/awk -f BEGIN { print "File\tOwner" }{ print $8, "\t", $3}END { print " - DONE -" }

In its simplest usage awk is meant for processing column-oriented text data, such as tables, presented to it on standard input. The variables $1, $2, and so forth are the contents of the first, second, etc. column of the current input line. For example, to print the second column of a file, you might use the following simple awk script:

awk < file '{ print $2 }'

This means "on every line, print the second field". 

By default awk splits input lines into fields based on whitespace, that is, spaces and tabs. You can change this by using the -F option to awk and supplying another character. For instance, to print the home directories of all users on the system, you might do

awk < /etc/passwd -F: '{ print $6 }'

since the password file has fields delimited by colons and the home directory is the 6th field. 

Awk is a weakly typed language; variables can be either strings or numbers, depending on how they're referenced. All numbers are floating-point. So to implement the fahrenheit-to-celsius calculator, you might write

awk '{ print ($1-32)*(5/9) }'

which will convert fahrenheit temperatures provided on standard input to celsius until it gets an end-of-file.  

echo 5 4 | awk '{ print $1 + $2 }'prints 9, whileecho 5 4 | awk '{ print $1 $2 }'prints 54. Note thatecho 5 4 | awk '{ print $1, $2 }'prints "5 4".

awk has some built-in variables that are automatically set; $1 and so on are examples of these. The other builtin variables that are useful for beginners are generally NF, which holds the number of fields in the current input line ($NF gives the last field), and $0, which holds the entire current input line. 

You can make your own variables, with whatever names you like (except for reserved words in the awk language) just by using them. You do not have to declare variables. Variables that haven't been explicitly set to anything have the value "" as strings and 0 as numbers.

For example, the following code prints the average of all the numbers on each line:

awk '{ tot=0; for (i=1; i<=NF; i++) tot += $i; print tot/NF; }'awk '{ tot += $1; n += 1; }  END { print tot/n; }'

Note the use of two different block statements. The second one has END in front of it; this means to run the block once after all input has been processed. 

You can also supply regular expressions to match the whole line against:

awk ' /^test/ { print $2 }'

The block conditions BEGIN and END are special and are run before processing any input, and after processing all input, respectively. 

awk supports loop and conditional statements like in C, that is, for, while, do/while, if, and if/else.

awk '{ for (i=2; i<=NF; i++) printf "%s ", $i; printf "\n"; }'

Note the use of NF to iterate over all the fields and the use of printf to place newlines explicitly. 

finding everything within the last 2 hours:

awk -vDate=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' access_log

Note: date is stored in field 4

To find something between 2-4 hrs ago:

awk -vDate=`date -d'now-4 hours' +[%d/%b/%Y:%H:%M:%S` -vDate2=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date && $4 < Date2 {print Date, Date2, $4} access_log'

The following will show you the IPs of every user who requests the index page sorted by the number of hits:

awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ }    END { for (i in ipcount) {        printf "%15s - %d\n", i, ipcount[i] } }' logfile.log

$7 is the requested url. You can add whatever conditions you want at the beginning. Replace the '$7 == "/" with whatever information you want.

If you replace the $1 in (ipcount[$1]++), then you can group the results by other criteria. Using $7 would show what pages were accessed and how often. Of course then you would want to change the condition at the beginning. The following would show what pages were accessed by a user from a specific IP:

awk -F'[ "]+' '$1 == "1.2.3.4" { pagecount[$7]++ }    END { for (i in pagecount) {        printf "%15s - %d\n", i, pagecount[i] } }' logfile.log

You can also pipe the output through sort to get the results in order, either as part of the shell command, or also in the awk script itself:

awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ }    END { for (i in ipcount) {        printf "%15s - %d\n", i, ipcount[i] | sort } }' logfile.log

Example how to remove duplicate lines in text file

awk '!x[$0]++' [text_file_name]

There are only a few commands in AWK. The list and syntax follows:

• if ( conditional ) statement [ else
• statement ]
• while ( conditional ) statement
• for ( expression ; conditional ; expression )
• statement
• for ( variable in array ) statement
• break
• continue
• { [ statement ] ...}
• variable = expression
• print [ expression-list ] [ > expression ]
• printf format [ , expression-list ] [ >
• expression ]
• next
• exit

Example:

#!/bin/awk -f BEGIN {# Print the squares from 1 to 10 the first wayi=1;while (i <= 10) {printf "The square of ", i, " is ", i*i;i = i+1;}# do it again, using more concise codefor (i=1; i <= 10; i++) {printf "The square of ", i, " is ", i*i;}# now end exit; }

abbreviation:

• NF : number of field 
• NR : number of record
• FS : field ssparator
• RS: record separator FS="\n"
• ORS : output record separator ORS="\r\n"
• FILENAME 

http://dedetoknotes.blogspot.co.id/2016/03/linux-awk.html

References:  

• http://www.hcs.harvard.edu/~dholland/computers/awk.html
• http://stackoverflow.com/questions/7706095/find-entries-in-log-file-within-timespan-eg-the-last-hour
• http://www.grymoire.com/Unix/Awk.html
• http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs 
• http://stackoverflow.com/questions/11532157/unix-removing-duplicate-lines-without-sorting 

How to remove fglrx and replacing with AMD/ATI in Debian Jessie

"The fglrx driver is incompatible with the GNOME desktop released as part of Debian 8 "Jessie", as it does not support the EGL interface (release notes). It is recommended to use the free radeon driver instead."

After installing fglrx, I can not enter Gnome. I share how to remove fglrx driver and resintall AMD/ATI in Debian Jessie.

1. Remove all fglrx driver
   # apt-get remove fglrx-driver fglrx-atieventsd libfglrx
2. Add this repository into /etc/apt/sources.list
   deb http://httpredir.debian.org/debian/ jessie main contrib non-free
3. Install or reinstall firmware
   # apt-get install firmware-linux-free firmware-linux-nonfree
   # apt-get install --reinstall firmware-linux-free firmware-linux-nonfree 
4. Install or reinstall radeon driver (see https://wiki.debian.org/AtiHowTo for your system)
   # apt-get install xserver-xorg-video-radeon libdrm-radeon1 radeontool
   # apt-get install --reinstall xserver-xorg-video-radeon libdrm-radeon1 radeontool
5. Replace /etc/X11/xorg.conf with your working configuration. In my case /etc/X11/xorg.conf.original-0.
6. You can start your Gnome
   # startx

Woking at Asus K45DR

• A8-4500M 
• AMD Radeon® Mobility™ HD7640G + HD 7470M Dual Graphics with 1GB DDR3 VRAM

Reference:

• https://wiki.debian.org/AtiHowTo
• https://wiki.debian.org/ATIProprietary

Blocking IP address that repeatedly blocked by fail2ban

Find IP address that repeatedly blocked by fail2ban:

# awk '($(NF-1) = /Ban/){print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort -n
...
3 212.83.190.184
3 221.203.142.71
3 74.208.10.191
5 183.3.202.105

Note: the variable NF equals the number of fields in each row of the logfile. So $NF is the value of the last field. Add iptables rules to drop incoming packages from particular IP address

# iptables -I INPUT -s 183.3.202.105 -j DROP

or you just want to drop access to your sshd

# iptables -I INPUT -p tcp -s  221.203.142.71 --dport ssh -j DROP

You can write a bash script to find IP address

# vi awkfindipfail2ban.sh
#!/bin/bash
awk '($(NF-1) = /Ban/) { print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort -n

Make it executable

# chmod 744 awkfindipfail2ban.sh

Run it

# ./awkfindipfail2ban.sh

Other way to search brute force attempt is search in authentication log file

# awk '/Failed password/ { print "Date: "$1" "$2" "$3"\tUsername: "$9"\t\tClient IP: "$11 }' /var/log/auth.log

You can also use grep

# grep "$(date|awk '{print $2" "$3}')" /var/log/auth.log|grep -E '(BREAK-IN|Invalid user|Failed|refused|su|Illegal)'

For this case:

      1 104.255.67.202
      1 109.161.202.72
      1 115.79.56.178
      1 125.212.232.119
      1 193.201.227.10
      1 193.201.227.18
      1 193.201.227.200
      1 193.201.227.68
      1 195.154.51.132
      1 202.99.172.155
      1 222.186.21.74
      1 27.255.81.142
      1 42.112.249.111
      1 45.32.61.182
      1 5.79.205.171
      1 59.47.5.239
      1 74.208.46.187
      1 82.165.151.8
      2 222.186.21.143

You may see the brute force using 4 different IP (193.201.227./24). For this case you can block them all with this command

# iptables -I INPUT -p tcp -s  193.201.227.0/24 --dport ssh -j DROP

No worries, your users still able to use your other service such as www in case those IP above used by legitimate user.

As your iptables rules increasing, your server performance may degrade. You can move your blocking rules into ipset. 

1. create your ipset rules for example mynetrules.
   # ipset create mynetrules hash:net
2. populate your ipset rules
   # ipset add mynetrules 212.83.190.184 
   # ipset add mynetrules 183.3.202.114 
   # ipset add mynetrules 221.203.142.71 
   # ipset add mynetrules 183.3.202.105
   # ipset add mynetrules 183.3.202.112
   # ipset add mynetrules [u18576666.onlinehome-server.com]
   # ipset add mynetrules 212.129.15.239
   # ipset add mynetrules 212.129.56.65 
   # ipset add mynetrules 222.186.21.143 
   # ipset add mynetrules 193.201.227.0/24
   # ipset add mynetrules 125.88.177.111
   # ipset add mynetrules [u19026996.onlinehome-server.com] 
   # ipset add mynetrules 183.3.202.88
3. add your ipset rules into first line in iptables rules
   # iptables -I INPUT -m set --match-set mynetrules src -j DROP
   or just to block access to your ssh
   # iptables -I INPUT -p tcp --dport 22 -m set --match-set mynetrules src -j DROP
   or you want to block multiple port
   # iptables -I INPUT -p tcp --match multiport --dports 80,443 -m set --match-set mynetrules src -j DROP

Now you may see your iptables more simple:

# iptables -L 

Chain INPUT (policy ACCEPT) target     prot opt source               destination DROP       all  --  anywhere             anywhere             match-set mynetrules src...

To list your ipset rules i.e. mynetrules

# ipset list mynetrules 

Name: mynetrules Type: hash:net Revision: 5 Header: family inet hashsize 1024 maxelem 65536 Size in memory: 17176 References: 1 Members: 183.3.202.88 183.3.202.114 193.201.227.0/24 ... 

Additional script to search IP banned at least twice: 

#!/bin/bash
echo "These IP banned at least twice"
awk '($(NF-1) = /Ban/) { print $NF}' /var/log/fail2ban.log | sort | uniq -c | {
  while read -r line1 line2
  do
    if [ "$line1" -ge 2 ]; then
      echo "$line1 $line2"
    fi
  done
}
echo "These IP range banned at least twice"
awk '($(NF-1) = /Ban/) { print $NF}' /var/log/fail2ban.log | awk 'BEGIN{FS="."} ; { printf("%s.%s.%s.0/24\n",$1,$2,$3)}' | sort | uniq -c | {
  while read -r line1 line2
  do
    if [ "$line1" -ge 2 ]; then
      echo "$line1 $line2"
    fi
  done
}

Interesting Command:

• # awk '/authentication failure/ { print $NF }' /var/log/auth.log | sort | uniq -c
• # awk '/root/ && /sshd/ { print $0 }' /var/log/auth.log 

References:

• http://www.the-art-of-web.com/system/fail2ban-log/
• http://stackoverflow.com/questions/22298623/echo-results-from-a-grep-search-in-shell-script 
• http://unix.stackexchange.com/questions/3176/what-strings-should-i-look-for-in-var-log-auth-log