2

Currently I'm using python 3.4.3 and developing PyQt5 application.

In my app, there's a QThread, and some large object(100MB) is being (pickle) dumped by the thread.

However, dumping that object requires 1~2 seconds, and it blocks the main thread about 1~2 seconds because of GIL.

How can I solve this issue(non-blocking the main thread)?

I think that serializing my object to string takes time and it requires GIL, eventually blocks the main thread.(As I know, writing to file does not require GIL)

I'm thinking about using Cython, but since I'm the beginner in cython, I'm not sure whether or not using Cython will solve this issue.

Is there any way to work around this issue?

Edit: I tried multiprocessing module, but the intercommunication time (passing shared memory variables across processes) also takes about 1~2 seconds, which eventually gives no advantages.

asqdf
  • 289
  • 2
  • 9
  • Is it compulsory for your app to be multithread? You could get rid of GIL if you use multiproccess instead. – Ricardo Silveira Jul 31 '16 at 05:57
  • @RicardoSilveira I tried multiprocessing module, but it also takes 1~2 seconds to pass the shared memory variables across processes, which eventually gives no advantages. I should've mentioned it in my question. – asqdf Jul 31 '16 at 06:01
  • Maybe [multiprocessing.sharedctypes](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.sharedctypes) – J.J. Hakala Jul 31 '16 at 10:39
  • I think this depends on the contents of the object you want to pickle. If it's made up of a big tree of smaller Python objects it's going to be hard to avoid needing the GIL. If it's (for example) a large array of a single numeric data-type you can probably do quite well at avoiding the GIL. (Also, multiprocessing uses pickle to pass data between threads on Windows so that may not help much) – DavidW Aug 01 '16 at 21:21
  • You were right. I solved my issue by making my object more simple (array of simple dictionaries) and using `file.write` directly instead of using pickle. (`file.write` does not require GIL). – asqdf Aug 02 '16 at 05:11

1 Answers1

2

I solved my issue.

The solution was

  1. Making my object really simple. In my case, I converted my object to array of simple stringified dictionaries.

  2. I used file.write(stringified_dictionaries) directly instead of using pickle. This reduced time for serializing python object to string.

Since disk I/O does not require GIL in python, the only moment main thread blocked was the moment of converting my object, which was really short.

asqdf
  • 289
  • 2
  • 9