OCaml - Parmap executing Lwt threads hangs on the execution

Question

This is a follow up to this question: How to synchronously execute an Lwt thread

I am trying to run the following piece of code:

open Lwt
open Cohttp_lwt_unix

let server_content2 x  =
    "in server content x" |> print_endline ;
    Client.get (Uri.of_string ("http://localhost:8080/"^x)) >>= fun (_, body) ->
        (Cohttp_lwt.Body.to_string body) >|= fun sc -> sc
        ;;
let reyolo () =
    List.init 10 (fun i -> server_content2 (string_of_int i) ) ;;
let par () = 
    let yolo = reyolo () in
    "in par" |> print_endline;
    Parmap.pariter 
        ~ncores:4 
        (fun p ->  "before run" |> print_endline ; "content:"^(Lwt_main.run p) |> print_endline  ; "after run" |> print_endline ) 
        (Parmap.L yolo);;

par ()

I expected this to perform 10 remote connections. What I get is in par function Lwt_main.run seems to stuck before doing an actual remote call.

I doubt it might be of any significance but the server that suppose to respond is made in python and looks like this:

import subprocess
from bottle import run, post, request, response, get, route

@route('/<path>',method = 'GET')
def process(path):
    print(path)
    return "yolo"

run(host='localhost', port=8080, debug=True)

I can respond to a simpler example; I highly doubt this is minimal. Among other changes, are you able to remove usage of Parmap, replacing it with, perhaps, Unix.fork? Or nothing at all? — antron, Mar 22 '20 at 18:22
I slimmed down the example. About the second part, Parmap is the key element of this problem because I want to integrate remote connections to my existing program that already uses Parmap. Getting rid of Parmap would end up in much more work than finding/implementing other solutions for remote get/post (with sockets for example). — zajer, Mar 22 '20 at 19:22
What I would do here, is inline the [source code](https://github.com/rdicosmo/parmap/blob/master/src/parmap.ml) of Parmap by creating several local modules in my program, and copying the source of Parmap into them. I would then slim down the resulting combined program, until hopefully only the call to `fork` was left. I would also try to reduce the 10-element list to 2 or even 1. It's less to worry about, and if you look at Parmap, it behaves differently if the number of cores is less than the fan-out, and that's useful to eliminate, if it is not necessary. — antron, Mar 23 '20 at 04:33
Alternatively, someone freely familiar with Parmap may be able to answer this outright, but I would have to defer to such a person :) — antron, Mar 23 '20 at 04:35
However, I do see that all your calls to `server_content2` occur in the parent process. What if you stored functions in `yolo`, or arguments (`string_of_int i` or `i`), so that `server_content2` would only get called in each child process? — antron, Mar 23 '20 at 04:37
I think you are on the right track, if I move initialization of a remote call inside `Parmap.pariter` then some of the calls are getting through. I say some because it sometimes does not work. For example if I change the length of the `yolo` list to 100 then sometimes 4 calls are established, sometimes 70+ and sometimes all of them. — zajer, Mar 23 '20 at 13:28
@antron after some more tests I came to a conclusion that the second reason for hanging up of threads was due to the server misbehavior. Right now I am using JSON-RPC server made in c++ and nothing like this happens. So the answer for the original question that you provided was a correct one. If you post it as an answer I will mark it as the correct one. — zajer, Mar 25 '20 at 01:18

score 1 · Accepted Answer · answered Mar 25 '20 at 09:08

The issue is that the calls to server_content2, which start the requests, occur in the parent process. The code then tries to finish them in the child processes spawned by Parmap. Lwt breaks here: it cannot, in general, keep track of I/Os across a fork.

If you store either thunks or arguments in the list yolo, and delay the calls to server_content2 so that they are done in the child processes, the requests should work. To do that, make sure the calls happen in the callback of Parmap.pariter.

OCaml - Parmap executing Lwt threads hangs on the execution

1 Answers1