1

I'm using scipy.optimize.linprog to solve a bunch (~6000) of LP problems of about 300 parameters each. This code takes a while to run (like 30 min, and I want to run it hundreds of times), and I'm wondering if I could speed it up significantly by moving into a compiled language (preferably C++).

I know that the source code of many python modules is actually in C (not sure about scipy.optimize), which is why I'm unsure that I would see a boost in speed. Anybody got any tips?

Battery_Al
  • 779
  • 1
  • 5
  • 20
  • Solving LPs is more about algorithms than programming language and scipy's linprog is way worse than the open-source alternatives CLP/CBC, GLPK and lpsolve. For those one has to find a usable wrapper-lib (some candidates). But your case is somewhat special as each problem is easy (fraction of a second) and setup-time matters too (pulp + clp would be easy, but it's probably sub-optimal as it's file-based). Porting lingprog will be a mess! If this is for academics and you are able to get a license, use gurobi, cplex or mosek with their API. If not, try cylp and then some lpsolve/glpk-based stuff – sascha Aug 22 '17 at 09:21
  • https://stackoverflow.com/users/2320035/sascha So I'm thinking to code the entire process in C++ (loop containing the LP setup, LP solver, and then broadcasting solution into larger array) and using Clp as the solver. Then I'll use ctypes to pass the resulting array back into my larger python program. Let me know if any of that sounds dumb – Battery_Al Aug 22 '17 at 16:29
  • It sounds dumb (and i'm ignoring the C++ LP solver, followed by clp as solver). This is a lot of work. Just use cylp as clp wrapper. And not everything ported to C++ will speed things up. – sascha Aug 22 '17 at 16:44
  • Alright man, you know I meant that Clp would be the solver in my C++ code. I just meant that I wanted to put the LP setup into C++ to save more time. But you're right, looking at Cylp and it looks like a quick fix... would Cylp/Clp really be that much faster than scipy.optimize.linprog? – Battery_Al Aug 22 '17 at 16:55
  • You seem happy to do a lot of very complex c++ stuff, so just invest these few minutes to try the cylp approach (k; setup might be hurdle on some OS'). For bigger LPs the difference is a few orders of magnitude probably. But this is depending on so much stuff and your case is a bit different. So you need to try it. – sascha Aug 22 '17 at 17:05
  • https://stackoverflow.com/users/2320035/sascha I really appreciate your help - cylp is like 40x faster. I've got some quick trivia for you though: do you know of an easy way to suppress Clp's diagnostic prints? Every time I employ .primal() it prints three lines I don't need to see. I'm not sure how to stop that without going into Clp's source. – Battery_Al Aug 23 '17 at 23:57
  • I assume you got something like ```s = CyClpSimplex()```. Then do ```s.logLevel = 0``` before solving. (if that's not working as expected, there is not much you can do. the cbcModel for MIPs has some noisy-output-behaviour when using cutting-planes and sadly: there is not much development anymore in regards to cylp) – sascha Aug 24 '17 at 00:24
  • You're my hero. – Battery_Al Aug 24 '17 at 00:33
  • stackoverflow.com/users/2320035/sascha Hey so this has worked wonders on my mac, but now I've got this program in a docker container running an Ubuntu image, and when the code gets to the CyLP section (specifically when I'm declaring constraints), I get `ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()`. Have you used CyLP in ubuntu before? Do you have any ideas about fixing this? – Battery_Al Aug 25 '17 at 01:24
  • might just be my python version (2.7.12 in ubuntu vs 2.7.13 on mac) – Battery_Al Aug 25 '17 at 01:32
  • It's not python. Either numpy or you are using py3 somewhere without noticing (and yeah i used cylp on ubuntu with success). – sascha Aug 25 '17 at 01:33
  • Which version of python are you running? I'm trying to install 2.7.13 into my container, but it's kind of a pain. – Battery_Al Aug 25 '17 at 02:17
  • https://stackoverflow.com/users/2320035/sascha So I think it has to do with which version of numpy I'm using. If I use continuum's anaconda2 image and then easy_install cylp, it works and I get `FutureWarning: comparison to 'None' will result in an elementwise object comparison in the future.`. But if I then conda install numpy (thereby updating a bunch of crap, scipy included), my program gives me a `ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()`. I'm afraid this is a cylp bug that only shows up in the most recent numpy. – Battery_Al Aug 25 '17 at 18:37
  • It's not a bug, but a change in numpy (as your analysis describes). cylp is not much developed anymore and therefore you would need to change it yourself or use an older numpy. – sascha Aug 25 '17 at 18:42

0 Answers0