Mixing Python and Go

Question

I have been working on a library in Python and I would like to do some performance improvement.

Is it possible to write some code in Python and some code in Go, and pass data between them? And if it's possible, are there any examples on how to do this?

Like such:

# Python
def python_foo():
    data = {'foo': 'val', 'bar': [1, 2, 3]}
    go_process(json.dumps(data))


def python_got_data_from_go(data):
    # deal with data from Go


# Go
func go_process(json string) {
    // do some processing
    python_got_data_from_go(someData)
}

possible duplicate of [Writing a Python extension in Go (golang)](http://stackoverflow.com/questions/12443203/writing-a-python-extension-in-go-golang) — Fred Foo, Apr 26 '14 at 17:15
That question was asked almost two years ago, and a lot has happened in both the Go community and Python community since then. — Jones, Apr 26 '14 at 17:21
Please check https://blog.filippo.io/building-python-modules-with-go-1-5/ — Nikolay Shindarov, Mar 16 '21 at 14:18

score 10 · Accepted Answer · edited May 23 '17 at 12:16

10

You need a glue between them, for example C programming language or communication over network. Most painful solution if you mix https://docs.python.org/2/extending/extending.html with http://golang.org/cmd/cgo/ and good programming skills in C.

You might create server in python http://pymotw.com/2/socket/tcp.html and in go https://coderwall.com/p/wohavg and communicate between them.

Edit: see Writing a Python extension in Go (Golang) and Calling Python function from Go and getting the function return value

edited May 23 '17 at 12:16

Community

1
1

answered Apr 26 '14 at 17:11

BalticMusicFan

653
1
8
21

I think perhaps running a py server and go server is a good solution. This way heavy lifting can be handled by Go. I like the sound of this solution and I will try it. – Jones Apr 26 '14 at 17:23

Lie Ryan · Answer 2 · 2014-12-29T05:42:44.413

The easiest way to mix languages is to to communicate over a socket, such as TCP/IP or Unix domain socket or through a high level protocol like HTTP or XML-RPC. This comes with very high overhead though due to the request processing and serialization/deserialization to/from JSON/XML, which can be significant if you have to make lots of calls. Communicating over socket is usually best if the amount of workload per request is high.

If you are not willing to pay for the overhead of socket (say if you make thousands of requests back and forth between python and go per seconds), there are other solutions that may have lower overhead. You may be able to use shared memory in OSes that have them. Shared memory usually incurs much cheaper cost to access data, but it may incur boxing/unboxing cost from the shared memory structure to Python datatypes. Also, note that you may have to manage the locks yourself with this.

Unless you only make very small number of calls and they do not need to share state between calls I would not recommend communicating using the standard stdin/stdout.

The last alternative is to write Python Extension; I would not recommend this for the faint of heart.

score 2 · Answer 3 · answered Apr 26 '14 at 21:11

2

The simplest solution is to spawn a Go process from Python and make them communicate via the Go's process standard streams (os.Stdin and os.Stdout). You have to invent a protocol both sides agree on (looks like you've settled on JSON already) and not to forget to flush the stream after streaming a logically atomic request, from both ends.

This way your solution is (almost) cross-platform and very easy to set up.

answered Apr 26 '14 at 21:11

kostix

51,517
14
93
176

Another way to think of this solution is like calling a command from a shell script - you just spawn the sub-process do to some job, and read its output, potentially calling it many times if need be. – joshlf Apr 27 '14 at 06:40

Mixing Python and Go

3 Answers3