Example of using hylang with python multiprocessing

Question

I am looking for an example of using python multiprocessing (i.e. a process-pool/threadpool, job queue etc.) with hylang.

Kodiologist · Accepted Answer · 2021-08-26T02:00:24.833

2

The first example from the multiprocessing documentation can be literally translated to Hy like so:

(import multiprocessing [Pool])

(defn f [x]
  (* x x))

(when (= __name__ "__main__")
  (with [p (Pool 5)]
    (print (.map p f [1 2 3]))))

edited Aug 26 '21 at 02:00

answered Aug 03 '19 at 14:29

Kodiologist

2,984
18
33

Does that work for you? When I try running it I get an error that indicates the child processes are attempting to parse some Python as Hy: ```Traceback (most recent call last): File ".../bin/hy", line 8, in sys.exit(hy_main()) File "", line 1, in NameError: name 'hyx_from' is not defined``` – Mark Reed Aug 25 '21 at 23:36
@MarkReed The `import` syntax was for a previous version of Hy, contemporary for when I wrote this answer. I've updated it and confirmed that it works on Hy master. – Kodiologist Aug 26 '21 at 02:01
With Python 3.9.6 and Hy 0.20.0 or 1.0a3, that doesn't work. First, the import is missing a set of square brackets; my install wants`(import [multiprocessing [Pool]])`. Second, I still get the NameError for` hyx_from` when trying to execute the example. – Mark Reed Aug 26 '21 at 13:41
@MarkReed Neither 0.20.0 nor 1.0a3 is up-to-date with master. The change to import syntax happened since 1.0a3. – Kodiologist Aug 26 '21 at 15:49
Again, the `import` syntax is not the problem, the mismatch between Hy and Python source when launching a new process is. It seems to be a macOS-specific issue; under Linux (e.g. in the docker image the git repo builds) it works, but even with the latest and greatest from github, running natively on Catalina results in a long chain of `NameError: name 'hyx_from' is not defined` messages. Seems like an odd thing to be OS-dependent, though. – Mark Reed Aug 26 '21 at 19:39
@MarkReed Odd? This is exactly the kind of thing that you should expect to be OS-dependent because `multiprocessing` has to be implemented differently depending on what system calls are available. You can file a bug if you like, but currently we're not testing on Mac OS. – Kodiologist Aug 27 '21 at 00:16

Mark Reed · Answer 2 · 2021-08-27T01:07:03.783

Note that a straightforward translation runs into a problem on macOS (which is not officially supported, but mostly works anyway): Hy sets sys.executable to the Hy interpreter, and multiprocessing relies on that value to start up new processes. You can work around that particular problem by calling (multiprocessing.set_executable hy.sys_executable), but then it will fail to parse the file containing the Hy code itself, which it does again for some reason in the child process. So there doesn't seem to be a good solution for using multiprocessing with Hy running natively on a Mac.

Which is why we have Docker, I suppose.

Example of using hylang with python multiprocessing

2 Answers2