0

I'm using the following code to grab the url http://brandonhsiao.com/essays.html:

(defparameter *new-line* '(#\Return #\Newline))

(defun read-url (host port path)
  (let ((sock (usocket:socket-connect host port)))
    (format (usocket:socket-stream sock) "~A"
            (concatenate 'string
                         "GET " path " HTTP/1.1" *new-line*
                         "Connection: close" *new-line* *new-line*))
    (force-output (usocket:socket-stream sock))
    (do ((line
           (read-line (usocket:socket-stream sock) nil)
           (read-line (usocket:socket-stream sock) nil))
         (all ""))
      ((not line) all)
      (setf all (concatenate 'string all line *new-line*)))))

(print (read-url "brandonhsiao.com" 80 "/essays.html"))

This gives me a 400 Bad Request error, but when I visit http://brandonhsiao.com/essays.html with Firefox, everything is fine. What am I doing wrong?

  • Make sure that new-line is sent to the socket as CRLF (HTTP requires newlines to be exactly CRLF). Since you use `format` to write text to socket's stream, newlines might be converted to LF. You can use tcpdump or wireshark to see which bytes are sent to socket. – dmitry_vk Aug 02 '13 at 05:40
  • I'd recommend that you use the "drakma" package which is designed to do these kinds of things. It's available from Quicklisp. – Elias Mårtenson Aug 02 '13 at 13:58

1 Answers1

1

It looks like I needed to include the host.

(defparameter *new-line* '(#\Return #\Newline))

(defun read-url (host port path)
  (let ((sock (usocket:socket-connect host port)))
    (format (usocket:socket-stream sock) "~A"
            (concatenate 'string
                         "GET " path " HTTP/1.1" *new-line*
                         "Host: " host *new-line* *new-line*))
    (force-output (usocket:socket-stream sock))
    (do ((line
           (read-line (usocket:socket-stream sock) nil)
           (read-line (usocket:socket-stream sock) nil))
         (all ""))
      ((not line) all)
      (setf all (concatenate 'string all line " ")))))