Python too slow - Can I rewrite engine in lower level and cp scripts?

Question

So, first off. Here's my server engine. gilmud.py!

Ok, here's a shortened version of my previous novel sized post.

The link above is our python server engine for our MUD. Noting line 73-75, and 359 on

self.tickThread = threading.Thread(None, self.runTicks, None, ())
self.tickThread.start()

...

def runTicks(self):
    while self.running:
        time.sleep(.1)
        for thing in Thing.List.values():
            if thing:
                if "person" in thing.attrs:
                    if "spawner" in thing.attrs:
                        thing.tick()

you'll probably see the horrid method of giving what we need to be roughly 100 players and 2000 mobs/npcs 'life'. The tick() checks for things like whether they'll move or pick up and item or if they are in combat or being targeted, etc. Same goes for the players, minus a few automated things, of course.

Is there any way we could rewrite a portion, or all of this module in say, C++ to get some better performance? Currently our needed .1 second ticks are at about 3 seconds using python in the method we have now.

(Also, we've tried SEVERAL different threading types and stackless. Nothing did the trick).

Thanks in advance for the help! Any advice is welcome!

Answer 1

You have a lot of inefficiencies in your code.

Looking in just the Person.tick method:

replace

for spell in self.spellTimers:
            self.spellTimers[spell]['tick'] += 1

with

for spell_timer in self.spellTimers.itervalues():
            spell_timer['tick'] += 1

same thing with at least two fewer dict lookups per value

replace

for thing in Thing.List.values():
            if thing:
                if "person" in thing.attrs:
                    if "spawner" in thing.attrs:
                        thing.tick()

with

for thing in Thing.List.itervalues():
            if thing and "person" in thing.attrs 
                     and "spawner" in thing.attrs:
                    thing.tick()

(ok that one might not be much faster but I like it better)

get:

from spells import cast,spellmaker

out of the tick() method and into the top of the file

replace:

#spell cooldown timer
self.timers['cooldown'] -= 1
if self.timers['cooldown'] < 0:
    self.timers['cooldown'] = 0

with:

#spell cooldown timer
self.timers['cooldown'] = max(0, self.timers['cooldown']-1)

You do that one a lot. You have a lot of code like

    #Check if self is doing ranged attack, and then increase timer (ready their weapon)
    self.timers['ranged'] += 1
    if self.timers['ranged'] >= int((self.stats['dex']*-1.1)+60+2):
        self.timers['ranged'] = int((self.stats['dex']*-1.1)+60+1)
    if self.timers['ranged'] == int((self.stats['dex']*-1.1)+60):
        self.timers['ranged'] = int((self.stats['dex']*-1.1)+60+1)

the second if should be elif if the first condition is true the second condition can NEVER be true.

remember every time you access self.somedict['name'] you are doing two dict lookups (overhead could be even more but use that as a rule of thum). If you have a dictlookup on line after line you speed up your code by assigning it to a "temp" local var.

If you keep going through your code, you'll put in one tweek like one of the above that will bring things inline. Sorry I can't go through it all.

Answer 2

Since you have not posted a small number of lines of relevant Python code, one thing you should know about Python is it will let you write code in a C-style, when an equivalent Python style will do. Often times -- and again I have not seen you relevant code -- an older style of processing data instead of, for example, writing a list comprehension, will cost more time than using a more Pythonic style.

You can get help getting a Pythonic style -- and BTW I need plenty of help -- by posting a small example or examples and asking if there is a better way to speed things up. You would also want to instrument your code to capture times.

Answer 3

It would be possible to rewrite parts of it in C++ or similar, but if your loop is taking 3 seconds to process 2000 or so items then I think it's more likely you've got some serious issues with the way you've coded things rather than an inherent problem with python itself.

So, firstly I'd like to second the recommendation of profiling that was made in the other answer/it's comments - it's really the best way to tell exactly where all the time is disappearing to in your loop. There's some documentation on the profilers provided with python available here.

Secondly, if you're using the standard python distribution then I'd consider staying away from threads if possible. The default implementation of python has a global interpreter lock that prevents any interpreted python code from truly running in parallel, removing a lot of the advantages of bothering with threads in the first place (and in some cases actually making things slower)

Third, I can offer some small advice on the code you've posted. You'll still be best using a profiler to confirm my suspicions (and to find problem areas that I haven't noticed - it looks like you've got quite a lot of code there) but I'd say this would probably help at least:

In your for loop for thing in Thing.List.values(): you'd probably be better calling Thing.List.itervalues(). This will return an iterator over the values rather than allocating a full list of the values. Allocating the list is just un-necessary unless you're planning on adding or removing to the dictionary whilst iterating over it, and might not be too fast if there's 2000 or so items in the dict as you imply. This advice also applies to other times you're iterating over dictionaries in your code, assuming you're not planning on adding/removing entries while iterating (Use .iteritems() instead of .items(), .iterkeys() instead of .keys()).

You might also want to look into a different timing method than calling time.sleep every iteration. There's two problems I can see with it - mainly that time.sleep(.1) doesn't actually guarantee that the sleep will last .1 of a second, and also because it doesn't take into account the length of time that processing will take. For example, say your processing takes 0.05 seconds then you sleep for .1 second, then there's actually been .15 seconds between ticks - and this gap will only grow as processing becomes more intense. I don't have any specific suggestions, but you may at least want to take the processing time into account when passing a number into time.sleep

Finally, you might want to consider using the code review stack exchange at some point. I don't think anyone would want to review your entire game engine but it seems like it might be a good place to put up chunks of your code that you might want feedback on, or that you think could be improved.

Answer 4

You've got a lot of great specific feedback already so I'll only add an observation from a mudder's perspective. There are a couple of large mud projects that use Python without speed issues (for example Evennia), so I'm confident in saying you're better off refactoring the code you have than dropping down to C.