0

I am new to PYPY and trying to use it with large arrays being worked on in a large iteration for loop. What libraries can I use to optimize for large arrays and are there any other optimizing techniques I should be aware of?

PLEASE NOTE THE *****SIMPLE CODE***** BELOW does not show that the print1-print10 variables CHANGE ON EVERY ITERATION. I am trying to keep things simple given that my question is ONLY about optimization techniques.

The code below is simple: 10 large arrays get merged in a for loop 28,000 times and all the numbers that fall out 3 times over all 10 arrays are calculated on every iteration. This is super slow, would take hours, to loop to 28000. 

print1=range(0,200000)
print2=range(100000,325000)
print3=range(180000,300000)
print4=range(0,250000)
print5=range(0,100000)
print6=range(170000,300000)
print7=range(160000,300000)
print8=range(150000,300000)
print9=range(140000,300000)
print10=range(130000,300000)

task_id=0

for x in range(0, 28800):
    print x
    result= print1 + print2 + print3 + print4 + print5 + print6 + print7 + print8 + print9 + print10
    from collections import Counter
    countresult=Counter(result)

    u = [ m for m,n in countresult.iteritems() if n == 3 ]
    print str(u) + "this is countsum"
user3152377
  • 429
  • 1
  • 5
  • 17
  • ...what's the point of any of this? Why are you running the same calculation 28800 times? Is this supposed to be calculating how intervals overlap? Why are you trying to do that by building giant arrays with almost no information content? – user2357112 Nov 21 '17 at 20:19
  • Note that at least 70% of the time is spent in the print statement. The result ``u`` is a long list. – Armin Rigo Nov 22 '17 at 12:51
  • 1
    Your question is not very clear. Are you asking "how can I do the same thing better"? Or the much vaguer "can you explain how to optimize with pypy in general"? – Armin Rigo Nov 22 '17 at 12:53
  • @ArminRigo am asking the much vaguer question of how to optimize pypy in general. I simply want suggestions of libraries to look into or any other techniques that might be useful in speed up of large arrays. I would also consider other programming languages if someone felt pypy was not the best way to go. – user3152377 Nov 23 '17 at 00:49

0 Answers0