Logs extracting, limiting cpulimit

Question

I have the following script for extraction of logs. Need to make work it with cpulimit, to not overload the server. Can anyone help?

nice -20 zcat /var/detail-20150418.gz | sed '/./{H;$!d;};x;/46.241.178.96/!d' > /tmp/new.txt 
nice -20 sed -n '/Acct-Status-Type/,/Called-Station-Id/p' /tmp/new.txt > /tmp/new_2.txt
rm /tmp/new.txt
nice -20 sed '/Acct-Status-Type/{x;p;x;}' /tmp/new_2.txt > /tmp/new_3.txt
rm /tmp/new_2.txt
grep -v '\(Acct-Authentic\|Acct-Input-Octets\|Acct-Input-Gigawords\|Acct-Output-Octets\|Acct-Output-Gigawords\|Acct-Input-Packets\|Acct-Output-Packets\|Acct-Session-Time\)' /tmp/new_3.txt > /tmp/new_4.txt
rm /tmp/new_3.txt
sed 's/^[ \t]*//;s/[ \t]*$//' /tmp/new_4.txt | paste -s -d ',' | sed 's/,,/\n/g' | sed 's/^[,]*//;s/[,]*$//' > /tmp/new_5.txt 
rm /tmp/new_4.txt
sed 's/Acct-Status-Type = //' /tmp/new_5.txt | sed 's/User-Name = //' |sed 's/Event-Timestamp = //' | sed 's/Acct-Terminate-Cause = //' | sed 's/Framed-IP-Address = //' | sed 's/Called-Station-Id = //' > /tmp/new_6.txt
rm /tmp/new_5.txt
awk -F"," '{ print $3 "," $1 "," $2 "," $6 "," $5 "," $4}' /tmp/new_6.txt | sed -e 's/"//g' | sed -e 's/,,/,/g' > /tmp/log.txt
rm /tmp/new_6.txt

any news? nobody will help? – user37033 Jun 29 '15 at 11:19 — user37033, Jun 29 '15 at 11:19

glenn jackman · Answer 1 · 2015-07-03T12:13:26.693

0

Getting rid of all that disk IO will help a lot:

nice -20 sh  <<'END_SCRIPT' > /tmp/log.txt
    zcat /var/detail-${date}.gz | 
    sed '/./{H;$!d;};x;/46.241.178.96/!d' |
    sed -n '/Acct-Status-Type/,/Called-Station-Id/p' |
    sed '/Acct-Status-Type/{x;p;x;}' |
    grep -Ev '(Acct-Authentic|Acct-Input-Octets|Acct-Input-Gigawords|Acct-Output-Octets|Acct-Output-Gigawords|Acct-Input-Packets|Acct-Output-Packets|Acct-Session-Time)' |
    sed 's/^[ \t]*//;s/[ \t]*$//' | 
    paste -s -d ',' | 
    sed -e 's/,,/\n/g' \
        -e 's/^[,]*//;s/[,]*$//' \
        -e 's/Acct-Status-Type = //' \
        -e 's/User-Name = //' \
        -e 's/Event-Timestamp = //' \
        -e 's/Acct-Terminate-Cause = //' \
        -e 's/Framed-IP-Address = //' \
        -e 's/Called-Station-Id = //' |
    awk -F"," '{ print $3 "," $1 "," $2 "," $6 "," $5 "," $4}' | 
    sed -e 's/"//g' | 
    sed -e 's/,,/,/g' 
END_SCRIPT

All the commands except zcat can be consolidated into a single awk or perl script: I don't have the time to help you with that right now.

edited Jul 03 '15 at 12:13

answered Jun 29 '15 at 13:11

glenn jackman

238,783
38
220
352

./IP.sh: line 2: syntax error near unexpected token `(' ./IP.sh: line 2: `nice -20 (' – user37033 Jun 29 '15 at 13:45
I updated my answer 2 days ago. You should have received a notification about it. – glenn jackman Jul 02 '15 at 12:44
Thank you. Do I need to write smth else except? ./script.sh {date} Because I have the following message: sh: -c: option requires an argument – user37033 Jul 03 '15 at 12:07
my mistake, remove `-c` – glenn jackman Jul 03 '15 at 12:13

Logs extracting, limiting cpulimit

1 Answers1