1

This question will be rather general, as I haven't even gotten to coding yet and am just wondering if it is possible to achieve what I want with lwIP.

What I would like, is for my embedded STM32F769I-Disco board to call a website URL API, e.g. http://test.com/items/1, which returns JSON, which I would then like to parse.

Since lwIP implements a TCP/IP stack, this should theoretically be possible with it, or am I mistaken?

I haven't really found any examples that would do this, or maybe I don't even know how to search for it. Any helpful pointers?

The question is also, should I perhaps use a socket connection rather than trying to call an API url? If I decided to use something like nanopb, would I still need lwIP, or could I do without it?

xtrinch
  • 2,232
  • 1
  • 24
  • 46

2 Answers2

6

Whatever involves retrieving data from a server involves either using either TCP or UDP, so it's not only "theoretically possible", it's the actual way your browser receives data from a web server: by opening a TCP/IP connection to the server's port 80.

UDP is not practical for receiving data from the server, because it's connectionless, and doesn't guarantee the packets will be received in order, or received at all. With TCP, it's "all or nothing" - either you will receive all the packets in order, or the socket will be closed at some point as the sender's buffer gets filled with unacknowledged packets.

So HTTP will use TCP/IP, that's for sure, and there is several steps you will need to do:

  1. Use lwip to create a DNS lookup for your www.test.com website.
  2. Use lwip to open a TCP/IP connection to the IP address, probably port 80.
  3. Attach a recv callback function (among others) to lwip, which will be called by Lwip whenever a packet is received. You will also want to know if the socket gets closed during the conversation, so make sure you register all the callbacks.

  4. Send a HTTP request string to the port 80. You are basically sending a string of characters to the opened TCP socket. This might be something as simple as:

    GET /items/1 HTTP/1.1
Host: www.test.com
<crlf>

    The <crlf> is an empty line at the end, it marks the end of the HTTP request.

  5. Your recv function will accept incoming packets from the server, and should write them to a FIFO buffer for further processing. There is no guarantee you will get the entire HTTP response in a single packet, so don't expect to handle them in one pass, unless the payload is really short, there are no proxies in between, and you feel lucky in general.

  6. So, you will then need to parse the incoming data as it arrives. One of the HTTP headers will give you the length of the entire response, or you will need to handle chunked encoding. This is an example of a response you might get from the server (in one or more TCP packets):

    HTTP/1.1 200 OK
Date: Mon, 23 Aug 2017 22:38:34 GMT
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: 18

{ 
    "value":5, 
    "something_else": [0, 1, 2] 
}

    Note the empty new-line between the headers and the start of the payload.

  7. Once you parse all the headers and get to the JSON, then you will probably use an existing C library for parsing JSON, which will also likely require some work to get an understanding of. Replacing this final step with Protocol Buffers might be a sensible idea from the point of development (likely less work on the C side), but the protocol will be less portable than JSON (although certainly smaller).

vgru
  • 49,838
  • 16
  • 120
  • 201
  • That was a pretty great explanation! It seems to me that there isn't really much difference whether I go implementing a custom protocol for the socket or just use HTTP requests. Thank you, I was a little confused on the basic concepts. – xtrinch Nov 20 '17 at 10:53
  • 2
    @Nirri: you can definitely create a simpler protocol than HTTP, if you can just connect to a plain TCP/IP socket. It would be easier (from the embedded programming perspective) to have a message which starts with a certain cookie, length, then contains data, and ends with a crc, and avoid having to deal with various nuances of HTTP. But if your server side already runs a web service of some sort, then you probably need to go this route (maybe you can also find a lightweight HTTP stack on top of lwip, I don't know). – vgru Nov 20 '17 at 11:20
1

From browsing the ST web site it looks like what you're asking is definitely possible if you're willing to put some time into understanding the platform. ST seems to publish a software development stack that supports your platform called STM32CubeF7. If you scroll down that link to the "User Manuals" section you'll see "UM1713: Developing applications on STM32Cube with lwIP TCP/IP stack" which sounds exactly like what you're asking.

Taking a step back, as you mention in the question, REST-ful website APIs are typically implemented as HTTP URLs supporting various operations (GET, POST, etc.). However HTTP sits on top of TCP/IP in the protocol stack and uses TCP/IP connections to transfer data to/from remote endpoints. It doesn't seem like lwIP provides an HTTP client so you will need to use another library, however there are plenty of GitHub projects which do just this. I recently wrote a blog post and codecast on implementing a simple HTTP client in C++ that might provide some insight and help dissuade you from rolling your own.

To your last paragraph, when speaking of "sockets", people are typically referring to TCP/IP (or UDP/IP) sockets, so yes you would still need lwIP. Nanopb would just replace HTTP in this case.

v1bri
  • 1,398
  • 8
  • 13