2

I would like to implement a server with HTTP.jl and julia. After some computation the server would return a "large" file (about several 100 MB). I would like to avoid having to read all the file in memory and then send it to the client.

Some framework allow have a specific function for this (e.g. Flask http://flask.pocoo.org/docs/0.12/api/#flask.send_file) or allow to stream the content to the client (http://flask.pocoo.org/docs/0.12/patterns/streaming/).

Are one for these two options also available in HTTP.jl ? Or any other Julia web package?

Here is a test code which reads the file testfile.txt, but I want to avoid loading the complete file in memory.

import HTTP

f = open("testfile.txt","w")
write(f,"test")
close(f)

router = HTTP.Router()

function testfun(req::HTTP.Request)  
  f = open("testfile.txt")
  data = read(f)
  close(f)
  return HTTP.Response(200,data)
end

HTTP.register!(router, "GET", "/testfun",HTTP.HandlerFunction(testfun))

server = HTTP.Servers.Server(router)
task = @async HTTP.serve(server, ip"127.0.0.1", 8000; verbose=false)

sleep(1.0)

req = HTTP.request("GET","http://127.0.0.1:8000/testfun/")

# end server
put!(server.in, HTTP.Servers.KILL)

@show String(req.body)
Alex338207
  • 1,825
  • 12
  • 16

1 Answers1

1

You can use memory mapped IO like this:

function testfun(req::HTTP.Request)
    data = Mmap.mmap(open("testfile.txt"), Array{UInt8,1})
    return HTTP.Response(200,data)
end

data now looks like a normal byte array to julia, but is actually liked to the file, which might be exactly what you want. The file will be closed upon garbage collection - if you have many requests and no garbage collection is triggered, you might end up with a lot of open files. If your request takes quite long anyway, you might consider calling gc() at the begin of the request.

christof
  • 26
  • 2