0

Is there some simple GUI API in python to handle multi-processing in python? A GUI with multi-frame and each frame displays information of one process such that I can easily check status of each processing. That's all my needs.

My story is both long and stupid. If you got enough patience, please continue:

I'm writing a python program to collect data from tens of websites in parallel, processing them and then store the data into mysql. The data are later used as data set for machine learning research, which should be continous in time demension.

Firstly, I use threading modular to do this. But partly due to my poor programming skills, the program collapses a lot. Then I ask in stackoverflow and follow some advice to adopt a demon process by importing the subprocess modular to restart the program when it collapses. Unfortunatelly, it still collapses a lot and can't successfully recover each time. (Don't know why yet: the demon process runs but can't restart the target program)

I think the frequent collapsing of my program is partly due to exception of data fetched from different website and in the consequent storing data to mysql stage. I got no idea to solve this and now I try to use one process for each task which both fetches data from one websit and then stores them to mysql, such that I may find it easier to catch such exeption and improve stability of my program. Modular multiprocessing is my current choice.

That's my sad story. Thank for at least finishing reading it. Any suggestion is highly appreciated.

Deep_fox
  • 405
  • 2
  • 6
  • 14
  • 1
    Catch all exceptions and log them. – Peter Wood Feb 10 '15 at 10:04
  • Not that I know of. Fetching data from websites in parallel is well suited to threading; multiprocessing wouldn't help much in terms of performance (as you won't be bound by the CPU) and isn't going to be easier to code. Find a good design pattern for a multithreaded program and it shouldn't be much harder to integrate it into a GUI. You could also look into PyQt which has QThreads built in (but are not required for multithreading with PyQt). – 101 Feb 10 '15 at 10:24
  • @PeterWood I've tried to catch all the exception using the logging modular. But still failed--The program simply throw a error and no exception information catched... – Deep_fox Feb 10 '15 at 10:39
  • @figs As you metioned, multiprocessing is still bound by CPU. But my most important problem is it not easy to catch the errors in my program. That's why I now turn to multiprocessing. As for PyQT, I heard of it and it sounds a little more difficult for me since my only need is to check the status of each process in one frame of GUI. It's quite possible that finally I have to turn to PyQT, but temporally I'd like to look for some easier API--if it exists. – Deep_fox Feb 10 '15 at 10:45
  • Which gui are you using? It shouldn't be too hard to track down errors even if they are in other threads – 101 Feb 10 '15 at 10:53
  • @figs I don't use any GUI yet. Now looking for some simply GUI to implement my ideas. Difficulties of of program is that it simply exits with an error code like 'memory confliction' and without any exception. I just guess it may be caused in the fetch data/ storing data stage. – Deep_fox Feb 10 '15 at 11:02
  • @Deep_fox, you should be asking a question about the errors then, with more real detail. – Peter Wood Feb 10 '15 at 12:00
  • @PeterWood Thanks for your suggestion. I'll try to catch the errors. – Deep_fox Feb 10 '15 at 12:15

0 Answers0