3

Protagonists

  • The Admin
  • Pipes
  • The Cron Daemon
  • A bunch of text processing utilities
  • netstat
  • >> the Scribe

Setting

The Cron Daemon is repeatedly performing the same job where he forces an innocent netstat to show the network status (netstat -n). Pipes then have to pick up the information and deliver it to bystanding text processing utilities (| grep tcp | awk '{ print $5 }' | cut -d "." -f-4). >> has to scribe the important results to a file. As his highness, The Admin, is a lazy and easily annoyed ruler, >> only wants to scribe new information to the file. 

*/1 * * * * netstat -n | grep tcp | awk '{ print $5 }' | cut -d "." -f-4 >> /tmp/file

Soliloquy by >>

To append, or not append, that is the question: 
Whether 'tis new information to bother The Admin with 
and earn an outrageous Fortune,
Or to take Arms against `netstat` and the others, 
And by opposing, ignore them? To die: to sleep;

note by the publisher: For all those that had problems understanding Hamlet, like I did, the question is, how do I check if the string is already included in the file and if not, append it to the file?

Gilles 'SO- stop being evil'
  • 104,111
  • 38
  • 209
  • 254
Florian Feldhaus
  • 5,567
  • 2
  • 38
  • 46

2 Answers2

1

Unless you are dealing with a very big file, you can use the uniq command to remove the duplicate lines from the file. This means you will also have the file sorted, I don't know if this is an advantage or disadvantage for you:

netstat -n | grep tcp | awk '{ print $5 }' | cut -d "." -f-4 >> /tmp/file && sort /tmp/file | uniq > /tmp/file.uniq

This will give you the sorted results without duplicates in /tmp/file.uniq

urish
  • 8,943
  • 8
  • 54
  • 75
  • by the way, awesome way to form a question :-) – urish Sep 21 '13 at 10:55
  • 1
    It works for me. I slightly modified it by using `sort`s capabilities to only return unique items and output the result to a file: `netstat -n | grep tcp | awk '{ print $5 }' | cut -d "." -f-4 >> /tmp/file && sort -u -o /tmp/file /tmp/file` – Florian Feldhaus Sep 21 '13 at 11:11
  • @AlexChamberlain Since you want to sort the content of the existing file along with the output of netstat command. – urish Sep 21 '13 at 11:27
  • `netstat -n | grep tcp | awk '{ print $5 }' | cut ... | sort -u > /tmp/file`. Doh? – Kaz Oct 08 '13 at 18:23
1
What a piece of work is piping, how easy to reason about, 
how infinite in use cases, in bash and script, 
how elegant and admirable in action,
how like a vim in flexibility, 
how like a gnu!

Here is a slightly different take:

netstat -n | awk -F"[\t .]+" '/tcp/ {print $9"."$10"."$11"."$12}' | sort -nu | while read ip; do if ! grep -q $ip /tmp/file; then echo $ip >> /tmp/file; fi; done;

Explanation:

awk -F"[\t .]+" '/tcp/ {print $9"."$10"."$11"."$12}'

Awk splits the input string by tabs and ".". The input string is filtered (instead of using a separate grep invocation) by lines containing "tcp". Finally the resulting output fields are concatenated with dots and printed out.

sort -nu

Sorts the IPs numerically and creates a set of unique entries. This eliminates the need for the separate uniq command.

if ! grep -q $ip /tmp/file; then echo $ip >> /tmp/file; fi;

Greps for the ip in the file, if it doesn't find it, the ip gets appended.

Note: This solution does not remove old entries and clean up the file after each run - it merely appends - as your question implied.

Kai Sternad
  • 22,214
  • 7
  • 47
  • 42