Scripting an HTTP header request with netcat

Question

I'm trying to play around with netcat to learn more about how HTTP works. I'd like to script some of it in bash or Perl, but I've hit upon a stumbling block early on in my testing.

If I run netcat straight from the prompt and type in a HEAD request, it works and I receive the headers for the web server I'm probing.

This works:

    [romandas@localhost ~]$ nc 10.1.1.2 80
    HEAD / HTTP/1.0

    HTTP/1.1 200 OK
    MIME-Version: 1.0
    Server: Edited out
    Content-length: 0
    Cache-Control: public
    Expires: Sat, 01 Jan 2050 18:00:00 GMT

    [romandas@localhost ~]$

But when I put the same information into a text file and feed it to netcat through a pipe or via redirection, in preparation for scripting, it doesn't return the headers.
The text file consists of the HEAD request and two newlines:

HEAD / HTTP/1.0

Sending the same information via echo or printf doesn't work either.

$ printf "HEAD / HTTP/1.0\r\n"; |nc -n 10.1.1.2 80
$ /bin/echo -ne 'HEAD / HTTP/1.0\n\n' |nc 10.1.1.2 80

Any ideas what I'm doing wrong? Not sure if it's a bash problem, an echo problem, or a netcat problem.

I checked the traffic via Wireshark, and the successful request (manually typed) sends the trailing newline in a second packet, whereas the echo, printf, and text file methods keep the newline in the same packet, but I'm not sure what causes this behavior.

FYI, the \r\n in the printf is purposeful; I switched between \r and \n in various combinations to see if it changed anything -- to no effect. — romandas, Mar 13 '09 at 13:30

score 27 · Accepted Answer · edited Feb 28 '23 at 04:06

27

You need two pairs of "\r\n", and also to tell netcat to wait for a response.

The following or something similar should work:

printf "HEAD / HTTP/1.0\r\n\r\n" | nc -n -i 1 10.1.1.2 80

edited Feb 28 '23 at 04:06

informatik01

16,038
10
74
104

answered Mar 13 '09 at 13:34

moonshadow

86,889
7
82
122

This doesn't work for me. I copied your line directly, but the connection still terminates before transmitting the header. – romandas Mar 13 '09 at 13:50
Then the problem is at your server end. printf "HEAD / HTTP/1.0\r\n\r\n" | nc www.toothycat.net 80 works here. – moonshadow Mar 13 '09 at 13:52
The server isn't terminating the request; it's initiated by the client. Wireshark shows my client sending a FIN packet right after the HEAD request packet, then the server ACKs and terminates gracefully. Not sure what causes the difference. What version of netcat are you using? – romandas Mar 13 '09 at 13:58
2

1.10-33 Oh, try adding a -q 10 to make netcat wait for a response after it gets the EOF on STDIN. – moonshadow Mar 13 '09 at 14:00
1

Ah, that did it. I'm using GNU netcat (v1.84), so the option is -i 1 (to wait between successive lines of input). That caused the second newline to be sent later, and produced the desired HEAD response. Edit that into your answer, and I'll go ahead and accept it. :) Thanks! – romandas Mar 13 '09 at 14:13
FYI, just the two newlines matter \n\n works (for me) just as well as \r\n\r\n. – romandas Mar 13 '09 at 17:28
The RFC requires CRLF pairs; best to be safe if you can. – moonshadow Mar 13 '09 at 18:50

score 9 · Answer 2 · edited May 09 '15 at 19:14

9

Another way is to use what is called the 'heredoc' convention.

$ nc -n -i 1 10.1.1.2 80 <<EOF
> HEAD / HTTP/1.0
>
> EOF

edited May 09 '15 at 19:14

Lightness Races in Orbit

378,754
76
643
1,055

answered Jan 23 '12 at 23:09

Bruce S

91
1
1

score 2 · Answer 3 · answered Nov 10 '15 at 18:08

2

Another way to get nc to wait for the response is to add a sleep to the input. e.g.

(printf 'GET / HTTP/1.0\r\n\r\n'; sleep 1) | nc HOST 80

answered Nov 10 '15 at 18:08

Jay Soffian

21
2

I am a fan of this approach because you have to specify an integer number of seconds with `nc`'s `-i` flag; however, with `sleep` you can specify decimals, and a much smaller interval seems to work for me (using a local server), like 0.01 seconds. – t354 Nov 11 '22 at 20:54

score 0 · Answer 4 · answered Jul 07 '16 at 05:37

0

You can use below netcat command to make your instance webserver:

MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print $1}')
while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo nc -l -p 80 ; done&

answered Jul 07 '16 at 05:37

Prashant Shetty

23
6

IPv4 IP which is used for default gateway (no matter what interface; Linux): `ip route show | grep -o 'src [\.0-9]\+' | uniq | cut -d ' ' -f2` – Ján Sáreník Jul 29 '16 at 08:53

score 0 · Answer 5 · answered Oct 05 '16 at 07:44

0

This line will also work as equivalent:

echo -e "HEAD / HTTP/1.1\nHost: 10.1.1.2\nConnection: close\n\n\n\n" | netcat 10.1.1.2 80

answered Oct 05 '16 at 07:44

Deming

1,210
12
15

Scripting an HTTP header request with netcat

5 Answers5

Linked

Related