3

I would like to use LuaSocket's HTTP module to download a large file while displaying progress in the console and later on in a GUI. The UI must never block, not even when the server is unresponsive during the transfer. Additionally, creating a worker thread to handle the download is not an option.

Here's what I got so far:

local io = io

local ltn12 = require("ltn12")
local http = require("socket.http")

local fileurl = "http://www.example.com/big_file.zip"
local fileout_path = "big_file.zip"

local file_size = 0
local file_down = 0

-- counter filter used in ltn12
function counter(chunk)
    if chunk == nil then
        return nil
    elseif chunk == "" then
        return ""
    else
        file_down = file_down + #chunk
        ui_update(file_size, file_down) -- update ui, run main ui loop etc.
        return chunk -- return unmodified chunk
    end
end

-- first request
-- determine file size
local r, c, h = http.request {
    method = "HEAD",
    url = fileurl
}
file_size = h["content-length"]

-- second request
-- download file
r, c, h = http.request {
    method = "GET",
    url = fileurl,
    -- set our chain, count first then write to file
    sink = ltn12.sink.chain(
        counter,
        ltn12.sink.file(io.open(fileout_path, "w"))
    )
}

There are a few problems with the above, ignoring error checking and hard-coding:

  1. It requires 2 HTTP requests when it is possible with only 1 (a normal GET request also sends content-length)
  2. If the server is unresponsive, then the UI will also be unresponsive, as the filter only gets called when there is data to process.

How could I do this making sure the UI never blocks?

user1973386
  • 1,095
  • 2
  • 10
  • 18

1 Answers1

1

There is an example on non-preemptive multithreading in Programming in Lua that uses non-blocking luasocket calls and coroutines to do a multiple parallel downloads. It should be possible to apply the same logic to your process to avoid blocking. I can only add that you should consider calling this logic from IDLE event in your GUI (if there is such a thing) to avoid getting "attempt to yield across metamethod/c-call boundary" errors.

Paul Kulchenko
  • 25,884
  • 3
  • 38
  • 56
  • I know I could do this with multithreading, but it's really overkill for something like this. I was thinking more in the lines of cURL's XFERINFO http://curl.haxx.se/libcurl/c/CURLOPT_XFERINFOFUNCTION.html . It's a simple callback function which gets called regardless if no data is being transfered (in the worst case at least once per second). However, I really don't want to add cURL as yet another dependency when I already depend on LuaSocket... – user1973386 Jun 24 '14 at 22:19
  • It's actually not that complex, so I don't think it would be overkill. It would only be using Lua coroutines, so it's not multithreading that involves OS threads, and is very lightweight. – Paul Kulchenko Jun 25 '14 at 03:42
  • you could of course also use lualanes for real multithreading, but coroutines should be easier. the only other way is to reimplement your whole application using some event main loop, but that's much more complicated. – nonchip Jun 27 '14 at 14:50