2

I am having difficulties parallelizing that part of code where new_text is of type unicode:

for old, new in self.replacements:
    line = pywikibot.replaceExcept(
        line, old, new, self.excsInside, self.site)
if new_text != entry.text:
    yield pywikibot.Page(self.site, entry.title)

The task looks to be easy with joblib or a process-pool, but there is new_text which is used outside of the loop. I have no idea of the equivalent of#pragama omp orderedor#pragma omp atomic, since there is no OpenMP wrapper for Python...

How do I determine what the value of new_next is going to be in the if statement if it's run in parallel?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
user2284570
  • 2,891
  • 3
  • 26
  • 74

1 Answers1

0

As it is inherently sequential, you can parallelize per line:

for line in new_text
    for old, new in self.replacements:
        line = pywikibot.replaceExcept(
            line, old, new, self.excsInside, self.site)

Ánd you parallelize the outer for loop (map with each replacement and then reduce by concatenation).

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
ratchet freak
  • 47,288
  • 5
  • 68
  • 106
  • 1
    I tried your code serially *(just pasted it inside replace.py)*... The program run during hours without any result, whereas it take 40min to finish normally. – user2284570 May 31 '14 at 08:41