Using Tools such as AWK, GREP, NAWK, XARGs on the Command Line

Extract the requests from an access_log that match the IPs from a list.

We have your common log formatted access_log. We wish to get all the requests that match the IPs from a list of IPs in a text file called ip_list.txt.

# awk '/.*/ {print "cat access_log | grep "$1" >> extract.txt "}' ip_list.txt | sh


Find out the Files That Contain Certain Text


It is common to want all the names of the files that contain a certain text. For instance, say inside a script directory you want to find all the scripts that contain a certain database name or a piece of text.

 find . -name '*.sh' | xargs grep -il '_KingKongDB'


Find out required libraries for C/C++ functions

Here we want to use the function sem_init, but we are not too sure of what libraries to link with. We can use 'nm' and 'grep' on the libraries (/usr/lib on linux).

# nm --print-file-name /usr/lib/* 2> /dev/null |grep sem_init

This command will return results with strings containing '/usr/lib/libpthread.a'. From this you know that you have to link with the option -pthread.


Replace a value in multiple files and write the output files to a new directory

In this example we have xml files such as Nokia-220s.xml, Nokia-2330.xml etc. We want to replace the value 'Spec-Handler' in these files with a new value called 'Nokia-WAP' and then write these files into a new directory called 'new'


# ls -l Nokia* |awk '{print "sed","'\''s/Spec-Handset/Nokia-WAP/g'\'' " $9 " >new/"$9}' |sh


Replace the newline ('\n') character at the end of a line with something else (a comma in this case)

cat addresses.txt | tr '\n' ','


Compare two files that are similar and list values in one file that do not appear in the other

cat file1 | awk '{print"echo " $2 " `grep -c "  $2" file2`"}' | sh | grep '0$'



Find the Min and Max in a list of values

For example if your file contents look like below
11
22
33
44

You can calculate min/max/average using nawk 

cat t | nawk '{print $1}' | nawk 'NR==1 {min=$1;max=$1;sum+=$1;next} {if ($1max) max=$1;sum+=$1} END{printf "min: %0.2f max: %0.2f avg: %0.2f\n", min,max,sum/NR}'

min: 11.00 max: 44.00 avg: 27.5

Misc

find . -type f | grep -v '\.svn' | xargs perl -pi -e 's/\r\n/\n/g' 

cat access_log | grep 217.124.181.30 |sed -n 's/.*\("[a-zA-Z][^"]*"\)$/\1/pg'

1 comment:

Brave Technologies said...

Thank you for this valuable information. Get your business to the next level in simple steps. We provides lowest price of erp Software for our clients erp providers in Chennai