How to send multiple data (conn:send()) with the new SDK (NodeMCU)

Question

I've been reading the NodeMCU documentation and several closed issues about the change of SDK that previouly allowed to send multiple data streams (acting like a queued net.socket:send).

It seems a huge debate grew here (#730) and there (#993) or even here (#999). However, I did not find any convincing example of a webserver code that would allow me to read multiple html files (e.g. head.html and body.html) to display a page. Here's the example from TerryE that I tried to adapt, but with no success:

srv=net.createServer(net.TCP)
srv:listen(80,function(conn)
    conn:on ("receive", function(sck, req)
        local response = {}

        local f = file.open("head.html","r")
        if f ~= nil then
            response[#response+1] = file.read()
            file.close()
        end

        local f = file.open("body.html","r")
        if f ~= nil then
            response[#response+1] = file.read()
            file.close()
        end

        local function sender (sck)
            if #response>0 then sck:send(table.remove(response,1))
            else sck:close()
            end
        end
        sck:on("sent", sender)
        sender(sck)
    end )
end )

When connecting to the ESP8266, nothing loads and I get no error from the lua terminal.

For your information, head.html contains:

<html>
<head>
</head>

And body.html contains:

<body>
<h1>Hello World</h1>
</body>
</html>

I am very new to NodeMCU, please be tolerant.

score 3 · Answer 1 · answered Mar 23 '16 at 01:01

Here is my solution without using tables, saving some memory:

function Sendfile(sck, filename, sentCallback)
    if not file.open(filename, "r") then
        sck:close()
        return
    end
    local function sendChunk()
        local line = file.read(512)
        if line then 
            sck:send(line, sendChunk) 
        else
            file.close()
            collectgarbage()
            if sentCallback then
                sentCallback()
            else
                sck:close()
            end
        end
    end
    sendChunk()
end


srv = net.createServer(net.TCP)
srv:listen(80, function(conn)
    conn:on("receive", function(sck, req)
        sck:send("HTTP/1.1 200 OK\r\n" ..
            "Server: NodeMCU on ESP8266\r\n" ..
            "Content-Type: text/html; charset=UTF-8\r\n\r\n", 
            function()
                Sendfile(sck, "head.html", function() Sendfile(sck, "body.html") end)
            end)        
    end)
end)

And this is for serving single files:

function Sendfile(client, filename)
    if file.open(filename, "r") then
        local function sendChunk()
            local line = file.read(512)
            if line then 
                client:send(line, sendChunk) 
            else
                file.close()
                client:close()
                collectgarbage()
            end
        end
        client:send("HTTP/1.1 200 OK\r\n" ..
            "Server: NodeMCU on ESP8266\r\n" ..
            "Content-Type: text/html; charset=UTF-8\r\n\r\n", sendChunk)
    else
        client:send("HTTP/1.0 404 Not Found\r\n\r\nPage not found")
        client:close()
    end
end


srv = net.createServer(net.TCP)
srv:listen(80, function(conn)
    conn:on ("receive", function(client, request)
        local path = string.match(request, "GET /(.+) HTTP")
        if path == "" then path = "index.htm" end
        Sendfile(client, path)
    end)
end)

S. Blanc · Accepted Answer · 2016-03-18T23:07:05.897

Thank you for the reply. I actually added the header you mentioned, I didn't know that was necessary and I also removed the sck argument in the sender function. My first code was actually working, I don't know what was wrong last time.

Anyway, it helped me understanding what was happening: the following code seems to concatenate the response array, since the event sent of the socket calls back the sender function (sck:on("sent", sender))

sck:send(table.remove(response,1))

In fact, table.remove(array, 1) returns the first item of the array, and removes this item of the array. Calling this line multiple times has the effect to read through it, item by item.

For the sake of simplicity, here is the code of a simple webserver able to serve multiple files:

header = "HTTP/1.0 200 OK\r\nServer: NodeMCU on ESP8266\r\nContent-Type: text/html\r\n\r\n"
srv=net.createServer(net.TCP)
srv:listen(80,function(conn)
    conn:on ("receive", function(sck, req)
        local response = {header}

        tgtfile = string.sub(req,string.find(req,"GET /") +5,string.find(req,"HTTP/") -2 )
        if tgtfile == "" then tgtfile = "index.htm" end
        local f = file.open(tgtfile,"r")
        if f ~= nil then
            response[#response+1] = file.read()
            file.close()
        else
            response[#response+1] = "<html>"
            response[#response+1] = tgtfile.." not Found - 404 error."
            response[#response+1] = "<a href='index.htm'>Home</a>"
        end
        collectgarbage()
        f = nil
        tgtfile = nil

        local function sender ()
            if #response>0 then sck:send(table.remove(response,1))
            else sck:close()
            end
        end
        sck:on("sent", sender)
        sender()
    end)
end)

This example was taken from this instructables and fixed to work with the new SDK (which do not allow multiple :send anymore). Please let me know if this code has some issues.

I don't know what is the size limit of the files though. Nevertheless, I manage to append more than 2Ko to the response variable and send it at once without any issue.

The beauty of Terry's initial example is that it works with an implicit loop with callbacks as you noted. It's taking one piece from the table (pls don't call it array), sends it and upon the sent-callback invokes the sender again to send the next piece until the table is empty. The only major issue with your code is that you send the same HTTP header regardless of the actual content. In the `file == nil` case your header should report HTTP 404 instead of HTTP 200. Hence, rather than adding `header` on line 5 you should do that conditionally within if/else. — Marcel Stör, Mar 25 '16 at 14:07
Also, http://lua-users.org/wiki/LuaStyleGuide -> Lua Idioms, you can use `if f then` instead of `if f ~= nil then`. Personally I'd never call a variable 'f' as I prefer expressive names but that's a matter of style. — Marcel Stör, Mar 25 '16 at 14:15
My main codicil is that it is worth the extra dozen lines of Lua to look ahead in the response array and count up how many fit in the maximum espconn buffer size, then marshal the first N buffers into a single send, e.g. `sck:send(table.concat({unpack(response,1,n)}))`. This might seem clunky, but most of this clunk is executed in code compiled directly from C so runs pretty efficiently and gives good packet sizes. — TerryE, Mar 26 '16 at 01:04
@TerryE what's the max espconn buffer size i.e. how many bytes? — Marcel Stör, Mar 26 '16 at 13:32

How to send multiple data (conn:send()) with the new SDK (NodeMCU)

2 Answers2

Linked