Benefit of converting Python method to C extension?

Question

A relatively simple question:
If I convert a CPU-bound bottleneck method from Python to a C extension (roughly implementing the same algorithm),

How much increase in speed, and performance should I expect?
What factors determine that?

UPDATE: People seemed to be complaining on the lack of specifics. I was mostly trying to understand what factors would make a piece of Python code a good candidate for being rewritten in C (i.e., when would porting to C actually give you a speed boost if the original Python is CPU-bound).

For specifics, this is the piece of code I'm looking at. Basically it's a recursive method that takes two lists of lists (a list of "columns", where each column contains possible values that could go in that column...basically, a schema), and seeing if it's possible to make less than n (usually 1) change(s) (where a change might be to add a new value to a column, add a new column, remove a column, etc.) such that there's some sequence of values (one value from each column) you could construct out of either schema. It's very similar in spirit to calculating the edit distance between to strings. Here's the code:

def CheckMerge(self, schemai, schemaj, starti, startj, \
               changesLeft, path):
#        if starti == 0 and startj == 0:
#            print '\n'
#            print schemai.schema
#            print ''
#            print schemaj.schema
    if starti == len(schemai.schema) and startj == len(schemaj.schema):
        return (True, path)
    if starti < len(schemai.schema):
        icopy = schemai.schema[starti]
    else:
        icopy = []
    if startj < len(schemaj.schema):
        jcopy = schemaj.schema[startj]
    else:
        jcopy = []
    intersect = set(icopy).intersection(set(jcopy))
    intersect.discard('')
    if len(intersect) == 0:
        if starti < len(schemai.schema) and \
            ('' in schemai.schema[starti] or changesLeft > 0):

            if not '' in schemai.schema[starti]:
                changesLeft -= 1
            changesCopy = list(path)
            changesCopy.append('skipi')
            result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj, \
                                     changesLeft, changesCopy)
            if result:
                return (result,steps)
            elif not '' in schemai.schema[starti]:
                changesLeft += 1

        if startj < len(schemaj.schema) and \
            ('' in schemaj.schema[startj] or changesLeft > 0):

            if not '' in schemaj.schema[startj]:
                changesLeft -= 1
            changesCopy = list(path)
            changesCopy.append('skipj')
            result,steps = self.CheckMerge(schemai, schemaj, starti, startj+1, \
                                     changesLeft, changesCopy)
            if result:
                return (result, steps)
            elif not '' in schemaj.schema[startj]:
                changesLeft += 1

        if changesLeft > 0:
            changesCopy = list(path)
            changesCopy.append('replace')
            changesLeft -= 1
            result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj+1, \
                                     changesLeft, changesCopy)
            if result:
                return (result, steps)

        return (False, None)
    else:
        changesCopy = list(path)
        changesCopy.append('merge')
        result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj+1, \
                                     changesLeft, changesCopy)
        if result:
            return (result, steps)
        else:
            return (False, None)

The speed increase may be from `1x`(= none) to about `100x`, depending on the data you are using and how the algorithm works. Your question is currently too vague for this site. You should include the python code you have and make specific question regarding that code. — Bakuriu, Mar 10 '13 at 08:13
Consider re-asking this question, after removing the code here... — pradyunsg, Mar 17 '13 at 03:57

score 1 · Answer 1 · answered Mar 10 '13 at 06:26

That solely and completely depends on your code. If some piece of your code is supported by the hardware, like, if you're computing the Hamming weight, doing AES encrption, calculating CRC, or have a vectorizable code, there are hardware instructions for them that boosts up the speed, and you can accesss them by C code but not python code.

score 0 · Answer 2 · answered Mar 10 '13 at 19:58

Python runs pretty fast, so you would need a distinct reason to convert a Python function to C, like to access hardware, which has already been mentioned. But, here is another reason.

Python (C Python) suffers from the Global Interpreter Lock (GIC) problem. Python threads cannot run simultaneously, only one at a time. So, you could put thread-specific code into C, which is not restricted by the GIC problem.

In general, if you believe your Python code is slow and it there is not a specific reason as you have mentioned in your post, then you may need to adapt to more Python-ic coding conventions, like list comprehensions and other features found in Python and not too many other languages.

My final comment is not a reflection on your code sample. Instead I am supplying it as the general wisdom that I've learned listening to a lot of Python presentations.

Benefit of converting Python method to C extension?

2 Answers2