51

I currently use Cython to link C and Python, and get speedup in slow bits of python code. However, I'd like to use goroutines to implement a really slow (and very parallelizable) bit of code, but it must be callable from python. (I've already seen this question)

I'm (sort of) happy to go via C (or Cython) to set up data structures etc if necessary, but avoiding this extra layer would be good from a bug fix/avoidance point of view.

What is the simplest way to do this without having to reinvent any wheels?

Community
  • 1
  • 1
tehwalrus
  • 2,589
  • 5
  • 26
  • 33

4 Answers4

47

Update 2015: possible as of Go 1.5 https://blog.filippo.io/building-python-modules-with-go-1-5/

with Go 1.5 you can build .so objects and import them as Python modules, running Go code (instead of C) directly from Python.

See also https://github.com/go-python/gopy

gopy generates a CPython extension module from a go package.

Colonel Panic
  • 132,665
  • 89
  • 401
  • 465
  • 4
    The github.com/go-python/gopy project makes this absurdly easy, creating all the necessary plumbing for you that the blog post mentions. – Chris Townsend Nov 10 '15 at 01:16
  • 4
    gopy is currently not compatible with Go>=1.6, so that was short lived. – mattmc3 Aug 03 '16 at 14:36
  • 1
    gopy was updated to support newer versions of Go. It is using the the "c-shared" build option and the pybindgen project – dre-hh May 28 '20 at 22:37
10

Unfortunately, this is not currently possible. Go can run C code (and that C code can then call back into Go), but the main function has to be in Go, so the Go runtime can set things up.

Russ Amos
  • 419
  • 3
  • 9
  • 1
    Perhaps it is possible to call go in a separate process trough xmlrpc then? – jsbueno Sep 16 '12 at 04:54
  • 1
    Thanks, this pretty much puts the idea to bed. The only way to do this, then, would be using something like `python_part_1 | go_program | python_part_2` using pipes (or intermediate files, or XMLRPC) to pass data? – tehwalrus Sep 16 '12 at 07:39
  • 2
    You can use sockets to do RPC. – Vladimir Matveev Sep 16 '12 at 08:33
  • 2
    It is fairly easy to embed Python into a C program. I wonder if you could then embed that C program into a Go program... The Python and Go interactions would have to go via the C layer. – Nick Craig-Wood Sep 16 '12 at 19:37
  • 2
    For communicating between Go and Python processes, you could Go run the Python program in a different process and use stdin, stdout to communicate between the processes (package os/exec). JSON would be better than XMLRPC for this because currently XML is a little harder to work with in Go than JSON is. – Matt Sep 17 '12 at 11:50
  • If you're using the stdin/out/err pipe to communicate it's better to not use any text-based format. Go for msgpack or something lighter, or even just serialize your structs over the pipe. Python has pack/unpack and Go has encoding/binary for this. – fiorix Apr 18 '14 at 01:52
  • 2
    This is now possible with Go 1.5 - see https://github.com/jbuberel/buildmodeshared for working examples. – Jason Buberel Aug 23 '15 at 15:12
3

I've written an extension to setuptools which allows you to write cpython extensions that interface with go: https://github.com/asottile/setuptools-golang

There's a couple example extensions here:

The neat thing is these can be installed just like any other pip package and support both cpython and pypy.

PEP 513 manylinux1 wheels can also be built to provide pre-built wheels via the setuptools-golang-build-manylinux-wheels tool.

The approach is nearly identical to the one in @ColonelPanic's answer but uses some additional tricks to enable python2 + python3 compatibility.

anthony sottile
  • 61,815
  • 15
  • 148
  • 207
2

There is a go-python package precisely to help you write Python extensions in Go:

this package provides an executable "go-python" which just loads "python" and then call python.Py_Main(os.Args). the rational being that under such an executable, go based extensions for C-Python would be easier to implement (as this usually means calling into go from C through some rather convoluted functions hops)

uriel
  • 1,467
  • 13
  • 14
  • 11
    From what I read, `go-python` helps you invoke Python from Go, not the other way around. – zneak Sep 17 '12 at 03:34
  • 1
    This is what I had seen, and discarded, precisely because it was the other way around as @zneak put it. – tehwalrus Sep 17 '12 at 19:12
  • 1
    I added a quote directly from the go-python readme that explains how to use it to build python extensions. – uriel Sep 19 '12 at 21:36