Python ctypes: WindowsError when using function pointer

Question

I'm trying to use Python's ctypes to work with a DLL, but I occasionally run into a problem when I try to call a function that's passed as a pointer to another function.

A little background...I'm trying to build a userspace filesystem using Dokan (version 0.6.0). Somewhat loosely stated, Dokan is basically FUSE for Windows. I've wrapped the dokan header file using ctypes (similar to pydokan). That header file contains the definition for a function pointer that looks like this

typedef int (WINAPI *PFillFindData) (PWIN32_FIND_DATAW, PDOKAN_FILE_INFO);

It also contains the prototype of another function

int (DOKAN_CALLBACK *FindFilesWithPattern) (
     LPCWSTR,
     LPCWSTR,
     PFillFindData,
     PDOKAN_FILE_INFO);

The corresponding ctypes definitions look like this

PFillFindData = ctypes.WINFUNCTYPE(ctypes.c_int,
                                   PWIN32_FIND_DATAW, 
                                   PDOKAN_FILE_INFO)

FindFilesWithPattern = ctypes.WINFUNCTYPE(ctypes.c_int,
                                          ctypes.c_wchar_p,
                                          ctypes.c_wchar_p,
                                          PFillFindData,
                                          PDOKAN_FILE_INFO)

The implementation of this latter function (FindFilesWithPattern) must call the FillFindData function that it is passed. A basic implemenation looks like this

def FindFilesWithPattern(self,
                         FileName,
                         SearchPattern,
                         FillFindData,
                         DokanFileInfo):
    if FileName == '\\':
        File = WIN32_FIND_DATAW(FILE_ATTRIBUTE_NORMAL | FILE_ATTRIBUTE_READONLY, 
                                FILETIME(1, 1),
                                FILETIME(1, 1), 
                                FILETIME(1, 1), 
                                0, 
                                len(self.HelloWorldText), 
                                0, 
                                0, 
                                'Hello_World.txt',
                                'Hello_~1.txt')
        pFile = PWIN32_FIND_DATAW(File)
        FillFindData(pFile, DokanFileInfo)
        return 0
    else:
        return -ERROR_FILE_NOT_FOUND

When this function gets called, I occasionally get the following error:

Traceback (most recent call last):
    File "_ctypes/callbacks.c", line 313, in 'calling callback function'
    File "src/test.py", line 385, in FindFilesWithPattern
        FillFindData(pFile, DokanFileInfo)
WindowsError: exception: access violation reading 0x0000000000000008

I am vexed. At first glance this looks like I'm trying to access memory that's out of range. But this error only occurs occasionally. Sometimes everything works fine, and the results come back as expected. (To clarify, when I say occasionally, I mean that the error occurs on some runs of the program and not on others. The error seems to occur or not occur consistently within a single run.)

I wondered then if perhaps I was getting an error code back rather than a memory address. I found here that if this was an error code, then it might indicate "not enough memory". When I look at the system monitor, this doesn't seem to be a problem though. I've tried running various memory profilers like Heapy and Meliae, but neither of them seem to work with Python 2.7 on Windows 64-bit.

My next best guess was that this is an issue using a 64-bit OS. Perhaps the type being used for the function pointer isn't sufficient to properly address it. After doing some Googling, it seems that others have had issues with using ctypes in Win64. I've built my Dokan library for a 64-bit architecture. Is there a problem with my python code then?

Any help would be very much appreciated. I've struggled with this for some time now.

A similar post can be found here. It doesn't seem terribly analogous though.

Note: In the python code you'll see some types that aren't defined here (e.g. PDOKAN_FILE_INFO). Those are either structures or pointers to structures that I didn't include for the sake of brevity.

Answer 1

How do you instantiate and use your callback? A callback must remain in scope for the lifetime it may be called. See this answer. These

---

Edit

Clarifying what I think the issue is...

I see you have to implement a DOKAN_OPERATIONS structure that implements callbacks like FindFilesWithPattern, then call DokanMain with that structure to mount the filesystem. When you fill out the structure you should be creating a type for the callback, then instantiating the callback with a Python function. Something like (pseudocode):

#Create the callback pointer type
PFINDFILESWITHPATTERN = ctypes.WINFUNCTYPE(ctypes.c_int,
                                          ctypes.c_wchar_p,
                                          ctypes.c_wchar_p,
                                          PFillFindData,
                                          PDOKAN_FILE_INFO) 

# Implement in Python
def FindFilesWithPatternImpl(...):
    # implementation

# Create a callback pointer object
FindFilesWithPattern = PFINDFILESIWTHPATTERN(FindFilesWithPatternImpl)

# Create the required structure listing the callback
dokan_op = DOKAN_OPERATIONS(...,FindFilesWithPattern,...)

# Register the callbacks
DokanMain(byref(dokan_op))

You must keep a reference to object dokan_op for the lifetime the callback could be used. If you implement mounting Dokan similar to below:

def mount():
    # Create structure locally
    dokan_op = DOKAN_OPERATIONS(...)
    # Spin off thread to mount Dokan
    threading.Thread(DokanMain,args=(byref(dokan_op),))

mount()

Once mount() returns dokan_op will be destructed. This is the scenario described by my original answer and would cause the error you are seeing. I'm theorizing your problem since I don't know what the rest of the code looks like, but I think the way you've implemented FindFilesWithPattern in Python is correct, esp. since you say it works intermittently. I've had the failure you're seeing before and the above scenario or something like it will cause the error you're seeing.

Hope this helps...