0

I would like to compile a Photoshop Filter plugin with OpenWatcom. The code already exists and works perfectly with Visual Studio.

The function looks like this:

extern "C" __declspec(export) void PluginMain(short selector, FilterRecordPtr pb, intptr_t *data, short *result);

It works perfectly in Visual C++. Note that there is explicitly no calling convention specified (Adobe does not define any calling convention in their SDK header files).

While it works in VC++, in OpenWatcom this does not work, and it seems like the calling convention is not correct. (I get weird behavior like corrupted stack, even with a "HelloWorld"-function).

At Wikipedia, I have looked at the table with the calling conventions: The table says that if no calling convention is specified, Watcom puts parameters in registers "EAX, EDX, EBX, ECX", stack filled right to left, and callee cleans the stack. But the table does not say what Microsoft compilers do if no calling convention is specified. I do think that Visual Studio uses a different calling convention as default. So I tried to specify __stdcall, __cdecl or __pascal in OpenWatcom, but it didn't help much.

One of my questions is, what exactly is Visual Studio's default calling convention if nothing is specified? What exactly will put into registers, in what direction on the stack, and who cleans the stack? I couldn't find this information anywhere.

If I would know the exact behavior, I might be able to use the "#pragma aux" functionality in OpenWatcom to enforce a custom calling convention?

Or do you know something else I can try to fix the calling convention in OpenWatcom?

Daniel Marschall
  • 3,739
  • 2
  • 28
  • 67
  • 1
    The default calling convention in VC++ is cdecl, although it can be changed through a command-line switch. The Watcom calling convention seems vaguely similar to VC++'s fastcall. Out of curiosity, why would you want to risk using a nowadays-esoteric compiler that adopts its own calling convention to build a plugin, so something where getting the ABI right is of paramount importance? – Matteo Italia Dec 20 '18 at 00:50
  • Thank you very much for the hint with cdecl. I didn't knew that, and I assumed that C++ had stdcall as default. (Can you post it as answer?) I have found out that the calling convention can be changed in OpenWatcom in Options -> C Compiler Switches -> Memory Model and Processor Switches. There, select "Pentium Pro stack-based calling" (default is "Pentium Pro register based calling"). Then, everything works :-) – Daniel Marschall Dec 20 '18 at 14:12
  • To your question: It has two reasons. The first reason is, that in my recreational time I love working and experimentalize with very old software, era 1990s - after all, these old things paved the ways for our modern computer technology. The second reason is, that I work on an OpenSource project, and in the building instructions I want to tell the people which compilers don't work at all, and which are working and how to configure them. – Daniel Marschall Dec 20 '18 at 14:12
  • Glad it helped, I'll move it to an aswer. I understand the fascination for "vintage" software completely, I suspected there was some motivation of that kind behind this. :-) – Matteo Italia Dec 20 '18 at 14:42

1 Answers1

2

(moving from a comment)

The default calling convention in Visual C++ is cdecl, although it can be changed through a command-line switch (/Gd, /Gr, /Gv, /Gz).

Reading around, it seems like Watcom has a similar set of options as well - -ecc should set it to use cdecl by default.

Matteo Italia
  • 123,740
  • 17
  • 206
  • 299