CLANG compiled MIDL RPC generated code incorrect function argument serialization

Question

Trying to run CLANG compiled, MIDL generated RPC client code, however it explodes on server side because the way MIDL generated client code serializes the function arguments in revers order.

MIDL is invoked as

midl MyRPCInterface.idl /env win32 /out"RPCx32" /char ascii7

Function looks like

int __cdecl f(int a, int b, int c);

RPC Client code for function looks like

int __cdecl f(int a, int b, int c)
{
    CLIENT_CALL_RETURN _RetVal;

    _RetVal = NdrClientCall2(
                  ( PMIDL_STUB_DESC  )&MyRPCInterface_StubDec,
                  (PFORMAT_STRING) & MyRPCInterface  __MIDL_ProcFormatString.Format[1110],
                  ( unsigned char * )&a);
    return ( char * )_RetVal. Simple;
}

MIDL generated client code assumes the arguments in memory are aligned [a][b][c], so it serializes them by taking the pointer to the first argument &a and assumes next variables follow.

This works very well for MSVC and BCC32, however for CLANG it does not. The arguments are in reverse order.

MSVC and BCC32

int* pA = &a;
int* pB = pA + 1;
int* pC = pB + 1;
int* pD = pC + 1;

CLANG

int* pD = &d;
int* pC = pD + 1;
int* pB = pC + 1;
int* pA = pB + 1;

After doing some low level debug, I can confirm that all 3 compilers pass the arguments in the correct order.

When MSVC and BCC32 use the arguments within the function body, they use them directly by referencing original stack addresses.

CLANG on the other side creates a copy of the variables and stores them in the reverse order in memory, and uses that new location when requesting variable addresses. Result is the function arguments are not correctly serialized, and server crashes with invalid arguments. Tried all optimization arguments

Question:

Is there a command line switch or keyword/pragma to tell CLANG to use variables directly from the stack and not copy them first? (why does it do, seems a waste of instructions)
Is there MIDL command line switch or keyword/pragma to tell it to reverse the variable order?

Thank you!

Here is simple test code to see order in memory:

#include <stdio.h>
#include <iostream>
#include <sstream>
#include <iomanip>
#include <cstdint>

void fdef(int a, int b, int c, int d)
{
    int* pA = &a;
    int* pB = &b;
    int* pC = &c;
    int* pD = &d;

    std::cout << " --------------------- " <<" Default call" << std::endl;
    std::cout << BytesToHexStr(&a, 12, true, "") << std::endl;
}

void __cdecl fcdecl(int a, int b, int c, int d)
{
    std::cout << " --------------------- " <<" __cdecl call" << std::endl;
    std::cout << BytesToHexStr(&a, 12, true, "") << std::endl;
}
void __stdcall fstdcall(int a, int b, int c, int d)
{
    std::cout << " --------------------- " <<" stdcall call"<< std::endl ;
    std::cout << BytesToHexStr(&a, 12, true, "") << std::endl;
}
void __pascal  fpascal(int a, int b, int c, int d)
{
    std::cout << " --------------------- " <<" pascal call"<< std::endl ;
    std::cout << BytesToHexStr(&a, 12, true, "") << std::endl;
}

int _tmain(int argc, _TCHAR* argv[]) 
{
    fdef(1, 2, 3, 4);
    fcdecl(1, 2, 3, 4);
    fstdcall(1, 2, 3, 4);
    fpascal(1, 2, 3, 4);
}

score 0 · Answer 1 · answered Jan 16 '18 at 22:39

Solved my issue by declaring RPC function having more than one parameter with _pascal calling convention. That forces MIDL compiler to generate code that is serializing each function parameter individually.

int _pascal f(int a, int b, int c);

The generated "C" code is does not compile directly using CLANG, however created a regex magic script to fix all code issues. MIDL generates a really bad/old code.

CLANG compiled MIDL RPC generated code incorrect function argument serialization

1 Answers1