Ctypes and Python - why the value is changed after just add one line for log?

Question

Current the c exposed the nvm_file , and python set the value and get value correct, but it changed after add the log info or other function?

C part

char *nvm_file = "./results/test01";
void get_file(void)
{
    printf("\nget file:%p %s\n", &nvm_file, nvm_file);
}

void set_file(char *file_name)
{
    nvm_file = file_name;
    printf("\nset file:%p %s\n", &nvm_file, nvm_file);
}

python part

def test_file():
    file_name = './results/plc_core_retain_test_result06'  
    libc.set_file(c_char_p(file_name.encode('utf-8')))
    log.debug("test")
    libc.get_file()

it work fine without "log.debug("test")",

set file:0x7fbc6a5d33c0 ./results/plc_core_retain_test_result06    
get file:0x7fbc6a5d33c0 ./results/plc_core_retain_test_result06

but the value was incorrect after add the log line

set file:0x7fe49aeed3c0 ./results/plc_core_retain_test_result06    
get file:0x7fe49aeed3c0 �/result

enter image description here

CristiFati · Answer 1 · 2022-04-15T13:56:09.697

Listing [Python.Docs]: ctypes - A foreign function library for Python.

What you have is Undefined Behaviour (argtypes and restype not being defined for functions). Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for the whole explanation.
What comes to my mind (although it's pure speculation) for your particular situation, is that the garbage collector has enough time to clean the (underlying) memory referenced by file_name between set_file and get_file calls, when log.debug is also called.

I modified your code to a working example.

dll00.c:

#include <stdio.h>

#if defined(_WIN32)
#  define DLL00_EXPORT_API __declspec(dllexport)
#else
#  define DLL00_EXPORT_API
#endif


char *nvm_file = "./results/test01";

#if defined(__cplusplus)
extern "C" {
#endif

DLL00_EXPORT_API void get_file();
DLL00_EXPORT_API void set_file(char *file_name);

#if defined(__cplusplus)
}
#endif


void get_file()
{
    printf("\nC: %s: %p [%s]\n", __FUNCTION__, &nvm_file, nvm_file);
}


void set_file(char *file_name)
{
    nvm_file = file_name;
    printf("\nC: %s: %p [%s]\n", __FUNCTION__, &nvm_file, nvm_file);
}

code00.py:

#!/usr/bin/env python

import ctypes as ct
import sys


DLL_NAME = "./dll00.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")


def main(*argv):
    dll00 = ct.CDLL(DLL_NAME)
    get_file = dll00.get_file
    get_file.argtypes = ()
    get_file.restype = None

    set_file = dll00.set_file
    set_file.argtypes = (ct.c_char_p,)
    set_file.restype = None

    file_name = b"./results/plc_core_retain_test_result06"
    get_file()
    set_file(file_name)
    get_file()

    # Another call to make sure it didn't work by coincidence
    set_file(b"aaa")
    get_file()


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q071880545]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2019\VC\Auxiliary\Build\vcvarsall.bat" x64 > nul

[prompt]> dir /b
code00.py
dll00.c

[prompt]>
[prompt]> cl /nologo /MD /DDLL dll00.c  /link /NOLOGO /DLL /OUT:dll00.dll
dll00.c
   Creating library dll00.lib and object dll00.exp

[prompt]>
[prompt]> dir /b
code00.py
dll00.c
dll00.dll
dll00.exp
dll00.lib
dll00.obj

[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe" code00.py
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] 064bit on win32


C: get_file: 00007FFD6AAF3000 [./results/test01]

C: set_file: 00007FFD6AAF3000 [./results/plc_core_retain_test_result06]

C: get_file: 00007FFD6AAF3000 [./results/plc_core_retain_test_result06]

C: set_file: 00007FFD6AAF3000 [aaa]

C: get_file: 00007FFD6AAF3000 [aaa]

Done.

score 0 · Answer 2 · answered Apr 15 '22 at 13:48

It's not the log.debug(), but anything. In the first case you do absolutely nothing between set and get, and simply you are lucky enough that the pointed location still contains the text.

But you shouldn't store that pointer, what you should do is making a copy of the text for yourself, and store that one. I have no means to test it now, but I think you could try something like this:

char *nvm_file = NULL;
void get_file(void)
{
    if(nvm_file)
        printf("\nget file:%p %s\n", &nvm_file, nvm_file);
    else
        printf("\nget file:<NULL>\n");
}

void set_file(char *file_name)
{
    nvm_file = realloc(nvm_file, strlen(file_name)+1);
    strcpy(nvm_file, file_name);
    printf("\nset file:%p %s\n", &nvm_file, nvm_file);
}

Mark Tolonen · Answer 3 · 2022-04-15T16:45:09.373

In the following code, file_name.encode('utf-8') is a byte string created as a temporary object. It is passed to c_char_p which creates a 2nd reference to the object, but both objects are dereferenced when the libc.set_file line completes, so the pointer stored by set_file points to deallocated memory by the time set_file runs. You just got lucky that the memory wasn't overwritten when there was no intervening debug line.

def test_file():
    file_name = './results/plc_core_retain_test_result06'  
    libc.set_file(c_char_p(file_name.encode('utf-8')))  # temp object
    log.debug("test")
    libc.get_file()

The code would appear to work if it was the following:

def test_file():
    file_name = b'./results/plc_core_retain_test_result06'.encode()
    libc.set_file(file_name)
    log.debug("test")
    libc.get_file() # file_name still referenced.

But in the above case file_name would be released after test_file returned. Any libc.get_file outside test_file would be referencing deallocated memory.

The correct solution would be to either store the string passed to set_file for the lifetime that get_file is used, or copy the string in the C code of set_file and let C manage the memory.

It's also advisable to set .argtypes and .restype for your ctypes-wrapped functions to help ctypes marshal the parameters correctly from Python to C without having to guess. Python can also check for correct types passed this way.

libc.get_file.argtypes = ()
libc.get_file.restype = None
libc.set_file.argtypes = c_char_p,
libc.set_file.restype = None

Thanks all of you, it works for me now. – LinusLing Apr 18 '22 at 12:17 — LinusLing, Apr 18 '22 at 12:17

Ctypes and Python - why the value is changed after just add one line for log?

3 Answers3