0

I am trying to run such sh script to gather geo information about Ip addresses. inputfile.txt is just a one column file with ip addresses in it.

Here is a sample of my input file:

213.100.122.171
213.100.126.188
213.100.129.186
213.100.18.247
213.100.23.238
213.100.26.151

Here is the code that I am using:

#!/usr/bin/env bash
output=outputfile.csv
for  i in $( cat "inputfile.txt"); 
do echo -e "$i,"$( geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $i | cut -d' ' -f6,8-9)"" >> $output;
done

When I run this file, the result is like this 

213.100.126.188
,LV, N/A, N/A,
213.100.129.186
,SE, N/A, N/A,
213.100.18.247
,LV, N/A, N/A,
213.100.23.238
,LV, N/A, N/A

I want them to be in each own line, like

213.100.126.188,LV, N/A, N/A,
213.100.129.186,SE, N/A, N/A,
213.100.18.247,LV, N/A, N/A,
213.100.23.238,LV, N/A, N/A,

Where i am wrong with my script?

Tom Fenech
  • 72,334
  • 12
  • 107
  • 141
jam
  • 65
  • 1
  • 11
  • You're wrong in many places: `for i in $(cat "inputfile.txt"); do` is wrong. What is the content of `inputfile.txt`? By the way, go and complain to the person who showed you this. Really do it. You're having `$(geoiplookup ...)` unquoted. This is bad (subject to pathname expansion). You're using an unquoted variable `$i` there too. You're opening and closing the file `outputfile.csv` for each ip, this is just uselessly resource consuming. Put your redirection at the end of the loop. Of course, none of these solve your problem! – gniourf_gniourf Feb 13 '15 at 12:50

3 Answers3

1

Using a for loop with cat is not a good way to process a file line by line. I would recommend changing your script to something like this:

#!/bin/bash

output=outputfile.csv    
while read -r line; do
    echo -e "$line, $( geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat "$line" | cut -d' ' -f6,8-9)"
done < inputfile.txt > "$output"

This will truncate the output file and write all of the output of the loop to it. Enclosing the command substitution as well as the parameter expansion in double quotes prevents issues with expansion of characters in the output of the command.

Tom Fenech
  • 72,334
  • 12
  • 107
  • 141
  • Unfortunately, the result is the same :/ – jam Feb 13 '15 at 13:12
  • As [gniourf_gniourf has mentioned above](http://stackoverflow.com/questions/28499766/avoid-newline-when-combining-two-variables-and-outputing-to-file#comment45319276_28499766), it would be useful to see the contents of your input file. Perhaps you could edit your question to include a sample of its contents – Tom Fenech Feb 13 '15 at 13:14
  • Added with example. there are only one column in file with ip addreses. Nothing more. – jam Feb 13 '15 at 13:19
  • @jam what system are you using? – Tom Fenech Feb 13 '15 at 13:21
0

Its most probably that application geoiplookup that produces the newline. So you have to pipe the output trough sed to remove it:

for  i in $( cat "inputfile.txt"); 
  do echo -e "$i,"$( geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $i | \
  cut -d' ' -f6,8-9 | sed ':a;N;$!ba;s/\n/)"" >> $output;
done
chaos
  • 8,162
  • 3
  • 33
  • 40
  • The newline after the command substitution is desired. It is the one from trying to read the file line by line using `for` with `cat` that is causing the problem. – Tom Fenech Feb 13 '15 at 12:52
  • 1
    Bash strips all trailing newlines with `$(...)`, so this can't be the answer. Try it: `a=$(echo test; echo; echo; echo; echo; echo); declare -p a`. You'll see no trailing newlines. Besides, since the `$(...)` is outside of the quotes, Bash will strip any newlines altogether. I can safely say that you're doubly wrong here. (But I'm not the downvoter!). – gniourf_gniourf Feb 13 '15 at 12:55
  • @TomFenech I have a file called `input` with 3 lines, doing `for i in $( cat "input"); do echo -e "-$i-"; done` prints all the lines without a newline after `$i`. If the file has `\n` at the end `echo -e` will interpret them as newlines, but not in our case. Do I understand something wrong? – chaos Feb 13 '15 at 13:17
0

Ok, i found the problem. The input file had an empty first line and the script somehow decided to work as I decribed. When I deleted that empty line, everything worked fine.

jam
  • 65
  • 1
  • 11