Using Tools such as AWK, GREP, NAWK, XARGs on the Command Line

Extract the requests from an access_log that match the IPs from a list.

We have your common log formatted access_log. We wish to get all the requests that match the IPs from a list of IPs in a text file called ip_list.txt.

# awk '/.*/ {print "cat access_log | grep "$1" >> extract.txt "}' ip_list.txt | sh


Find out the Files That Contain Certain Text


It is common to want all the names of the files that contain a certain text. For instance, say inside a script directory you want to find all the scripts that contain a certain database name or a piece of text.

 find . -name '*.sh' | xargs grep -il '_KingKongDB'


Find out required libraries for C/C++ functions

Here we want to use the function sem_init, but we are not too sure of what libraries to link with. We can use 'nm' and 'grep' on the libraries (/usr/lib on linux).

# nm --print-file-name /usr/lib/* 2> /dev/null |grep sem_init

This command will return results with strings containing '/usr/lib/libpthread.a'. From this you know that you have to link with the option -pthread.


Replace a value in multiple files and write the output files to a new directory

In this example we have xml files such as Nokia-220s.xml, Nokia-2330.xml etc. We want to replace the value 'Spec-Handler' in these files with a new value called 'Nokia-WAP' and then write these files into a new directory called 'new'


# ls -l Nokia* |awk '{print "sed","'\''s/Spec-Handset/Nokia-WAP/g'\'' " $9 " >new/"$9}' |sh


Replace the newline ('\n') character at the end of a line with something else (a comma in this case)

cat addresses.txt | tr '\n' ','


Compare two files that are similar and list values in one file that do not appear in the other

cat file1 | awk '{print"echo " $2 " `grep -c "  $2" file2`"}' | sh | grep '0$'



Find the Min and Max in a list of values

For example if your file contents look like below
11
22
33
44

You can calculate min/max/average using nawk 

cat t | nawk '{print $1}' | nawk 'NR==1 {min=$1;max=$1;sum+=$1;next} {if ($1max) max=$1;sum+=$1} END{printf "min: %0.2f max: %0.2f avg: %0.2f\n", min,max,sum/NR}'

min: 11.00 max: 44.00 avg: 27.5

Misc

find . -type f | grep -v '\.svn' | xargs perl -pi -e 's/\r\n/\n/g' 

cat access_log | grep 217.124.181.30 |sed -n 's/.*\("[a-zA-Z][^"]*"\)$/\1/pg'

No comments: