I don’t know much about Direct2D, heck I don’t know much about Direct3D. I’m not a graphics programmer, the only game I ever made was a Java version of Paratrooper. Regardless, its always useful to know more, so in order to have a little dabble and gentle easing in to this supposed successor of GDI, I thought I’d translate something that’s not too involved but still useful.
I’ve been messing with Pete’s Software renderer for ePSXe the PlayStation emulator so, since it uses DirectDraw, I decided to translate that. Now most of Microsoft’s newer native code API’s are targeted at C++, with bindings for C. It just so happens that this software renderer uses C, so no problem. I converted the initialization function, ran in it the debugger to make sure nothing went wrong and:
Direct2D – Run Time Check Error #0 – ESP Not Properly Saved
Oh, don’t think that was meant ot happen. Time for a standalone example, since ePSXe loads a lot of dlls.
#define WIN32_LEAN_AND_MEAN #define D2D_USE_C_DEFINITIONS #define CINTERFACE #include <d2d1.h> #include <stdio.h> #pragma comment(lib, "d2d1.lib") int __cdecl main() { ID2D1Factory* pFactory = NULL; D2D1_RENDER_TARGET_PROPERTIES renderProps; D2D1_HWND_RENDER_TARGET_PROPERTIES hwndRenderProps; D2D1_FACTORY_OPTIONS opts = {D2D1_DEBUG_LEVEL_INFORMATION}; D2D1_PIXEL_FORMAT pixFmt = {DXGI_FORMAT_UNKNOWN, D2D1_ALPHA_MODE_UNKNOWN}; HRESULT hr = S_OK; D2D1_SIZE_U targetSize; D2D1_SIZE_U pixSize; HWND hwndCon = GetConsoleWindow(); RECT winSize; ID2D1HwndRenderTarget* pHwndRenderTarget = NULL; hr = D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, IID_ID2D1Factory, &opts, (void**)&pFactory); GetClientRect(hwndCon, &winSize); pixSize.width = winSize.right; pixSize.height = winSize.bottom; renderProps.type = D2D1_RENDER_TARGET_TYPE_DEFAULT; renderProps.pixelFormat = pixFmt; renderProps.minLevel = D2D1_FEATURE_LEVEL_DEFAULT; renderProps.usage = D2D1_RENDER_TARGET_USAGE_NONE; renderProps.dpiX = renderProps.dpiY = 0.f; hwndRenderProps.hwnd = hwndCon; hwndRenderProps.pixelSize = pixSize; hwndRenderProps.presentOptions = D2D1_PRESENT_OPTIONS_NONE; if(FAILED(hr = ID2D1Factory_CreateHwndRenderTarget(pFactory, &renderProps, &hwndRenderProps, &pHwndRenderTarget))) { printf("Failed to create render target, err = %#x\n", hr); return 0; } pixFmt = ID2D1HwndRenderTarget_GetPixelFormat(pHwndRenderTarget); printf("HWND RT PixFmt: %lu, alpha: %lu\n", pixFmt.format, pixFmt.alphaMode); ID2D1HwndRenderTarget_Release(pHwndRenderTarget); ID2D1Factory_Release(pFactory); }
Right, tried again and
HwndRenderTarget GetPixelFormat crash
Huh!? Ok. So there’s definitely some problem with GetPixelFormat. If you’ve programmed for any length of time, you’ll recognise the first error above as meaning ‘the function prototype has the wrong number of parameters or wrong calling convention’. The docs for the function disagree that this is the problem since it takes no parameters (beside the interface pointer which is implicit for C++), and the calling convention is definitely stdcall since all Microsoft interface-based api’s are. So what gives?
For some reason which I can’t remember, I coverted it to use the C++ interface to see if that’d change anything, although there’s no way it could.
How did that happen? It works!
What? Something crazy is going on here. How did that work? All that changed was that the first parameter was passed implicitly. Actually, that wasn’t true at all. Oh, it did happen, but so did something else.
To go back to the C compiler/interface[1] for a minute, this is the generated code. It’s nothing special, exactly what you’d expect for a COM interface call taking no parameters:
; pixFmt = ID2D1HwndRenderTarget_GetPixelFormat(pHwndRenderTarget); 00411384 mov eax,dword ptr [ebp-6Ch] ; interface ptr 00411387 push eax ; pass interface as only parameter 00411388 mov ecx,dword ptr [ebp-6Ch] ; interface ptr 0041138B mov edx,dword ptr [ecx] ; vtable 0041138D mov eax,dword ptr [edx+0C8h] ; function to call 00411393 call eax ; call it
You’d expect the C++ code to look identical, and it mostly does. Except it also sneakily causes the function to allocate more stack space, and passes an extra argument to the function.
; pixFmt = pHwndRenderTarget->GetPixelFormat(); 00411384 lea eax,[ebp-0B4h] ; take address of something on the stack 0041138A push eax ; pass it as a parameter 0041138B mov ecx,dword ptr [ebp-6Ch] ; interface ptr 0041138E mov edx,dword ptr [ecx] ; vtable 00411390 mov eax,dword ptr [ebp-6Ch] ; interface ptr again 00411393 push eax ; pass interface as parameter 00411394 mov ecx,dword ptr [edx+0C8h] ; function to call 0041139A call ecx ; call it
Again, what’s going on here. Does GetPixelFormat take one or two parameters? Or maybe it’s actally a stdcall varags function that somehow cleans the stack itself? To the disassembly!
The implementation in question here[2] isn’t very long, but contains a lot of noise for our purposes. So I’ve trimmed out the bits we don’t need, the full code is below.
; D2DDCRenderTarget::GetPixelFormat: ... ... 729E9FF4 mov esi,dword ptr [ebp+8] ; first parameter, so it definitely takes at least one 729E9FF7 mov ecx,dword ptr [esi+8] ; It's a 12 byte structure at least 729E9FFA mov eax,dword ptr [ecx] ; One of it's members is a class 729E9FFC push edi 729E9FFD mov dword ptr [ebp-14h],ecx 729EA000 call dword ptr [eax+0Ch] ; Since it calls a thiscall function ... ... 729EA045 mov edi,dword ptr [ebp+0Ch] ; A-ha, a second argument so it takes at least two 729EA048 mov ecx,eax ; get some- 729EA04A mov eax,dword ptr [ecx] ; data into- 729EA04C mov ecx,dword ptr [ecx+4] ; the right registers 729EA04F mov dword ptr [edi],eax ; We already knew the argument was a pointer, but here's proof it's at least a DWORD 729EA051 mov dword ptr [edi+4],ecx ; And here's proof it's at least 2 dwords, so 8 bytes ... ... 729EA064 mov eax,edi ; edi has been unchanged since above so it returns the second argument 729EA066 pop edi 729EA067 pop esi 729EA068 leave 729EA069 ret 8 ; and here's final proof it takes two arguments
That seems pretty conclusive. It takes two parameters, the interface pointer and another pointer that it also returns. Since we know it returns a D2D1_PIXEL_FORMAT structure, and those are composed of two DWORD sized values, that’s what the second argument is a pointer to.
In short, the C interface to Direct2D is broken. It doesn’t account for the automagic argument passing of the C++ compiler which the definitions seem to rely on. It’s not specific to GetPixelFormat either, that’s just where I noticed it. All of the D2D functions that return structures seem to be affected, this includes render target functions like GetSize and GetPixelSize, and brush functions like ID2D1SolidColorBrush::GetColor.
Talking of ID2D1SolidColorBrush::GetColor, the compiler weirdness doesn’t end there. This function is like a red rag to a bull and causes the compiler to get stuck between both worlds. With the C interface, calling it produces some really wacky code:
; black = ID2D1SolidColorBrush_GetColor(pBrush); 00411409 mov eax,dword ptr [ebp-74h] ; Interface ptr 0041140C push eax ; Pass it as the 2nd parameter! 0041140D mov ecx,dword ptr [ebp-74h] ; Interface ptr 00411410 mov edx,dword ptr [ecx] ; Vtable 00411412 lea eax,[ebp-0F0h] ; Invent a stack parameter 00411418 push eax ; Pass it as the first parameter! 00411419 mov ecx,dword ptr [edx+24h] ; Function to call 0041141C call ecx ; call it
Even though it got the number of parameters right, the call would probably still crash since the parameters are passed in the wrong order. I’m not sure how you make code that messes up the compiler this badly, but Direct2D managed it.
All these problems are fixable, but require you to do that questionable practice of editing the headers. All that needs to be done is adding a pointer to the struct they return as the last parameter. The return value also needs updating to a pointer otherwise assigning the return value will produce incorrect values.
Not the greatest introduction I’ve ever had to an API.
Notes [1]: The disassembly in this post was produced by Visual Studio 2013, I tried all the code and variations in VS2008 as well and the relevant output was identical so either this isn’t a compiler bug, or a longstanding one.
[2]: There are other implementations of GetPixelFormat for different render target types, they all behave the same in regards to this post, MSDN’s definitions and the Direct2D headers.
[3]:
D2DDCRenderTarget::GetPixelFormat: 729E9FEB mov edi,edi 729E9FED push ebp 729E9FEE mov ebp,esp 729E9FF0 sub esp,14h 729E9FF3 push esi 729E9FF4 mov esi,dword ptr [ebp+8] 729E9FF7 mov ecx,dword ptr [esi+8] 729E9FFA mov eax,dword ptr [ecx] 729E9FFC push edi 729E9FFD mov dword ptr [ebp-14h],ecx 729EA000 call dword ptr [eax+0Ch] 729EA003 lea ecx,[ebp-10h] 729EA006 call FPUStateX87<0>::FPUStateX87<0> (729DDFF1h) 729EA00B lea eax,[ebp-0Ch] 729EA00E call FPUStateSSE<1>::FPUStateSSE<1> (729DE039h) 729EA013 cmp dword ptr [esi+1A0h],0 729EA01A jne D2DDCRenderTarget::GetPixelFormat+4Dh (729EA038h) 729EA01C cmp dword ptr [esi+0C0h],0 729EA023 jne D2DDCRenderTarget::GetPixelFormat+4Dh (729EA038h) 729EA025 mov ecx,dword ptr [esi+58h] 729EA028 mov eax,dword ptr [ecx+40h] 729EA02B mov edi,dword ptr [ebp+0Ch] 729EA02E mov dword ptr [edi],eax 729EA030 mov eax,dword ptr [ecx+44h] 729EA033 mov dword ptr [edi+4],eax 729EA036 jmp D2DDCRenderTarget::GetPixelFormat+69h (729EA054h) 729EA038 lea eax,[ebp-8] 729EA03B push eax 729EA03C lea eax,[esi+10h] 729EA03F push eax 729EA040 call DrawingContext::GetPixelFormat (729F8512h) 729EA045 mov edi,dword ptr [ebp+0Ch] 729EA048 mov ecx,eax 729EA04A mov eax,dword ptr [ecx] 729EA04C mov ecx,dword ptr [ecx+4] 729EA04F mov dword ptr [edi],eax 729EA051 mov dword ptr [edi+4],ecx 729EA054 lea esi,[ebp-10h] 729EA057 call CFloatFPU::~CFloatFPU (729DDFC5h) 729EA05C mov ecx,dword ptr [ebp-14h] 729EA05F mov edx,dword ptr [ecx] 729EA061 call dword ptr [edx+10h] 729EA064 mov eax,edi 729EA066 pop edi 729EA067 pop esi 729EA068 leave 729EA069 ret 8
I looked at the disassembly and this is produced for C, the [url=http://msdn.microsoft.com/en-us/library/windows/desktop/dd316814(v=vs.85).aspx]function is declared as taking no parameters[/url], so just the interface is pushed
pixFmt = ID2D1HwndRenderTarget_GetPixelFormat(pHwndRenderTarget);
00411384 mov eax,dword ptr [ebp-6Ch]
00411387 push eax
00411388 mov ecx,dword ptr [ebp-6Ch]
0041138B mov edx,dword ptr [ecx]
0041138D mov eax,dword ptr [edx+0C8h]
00411393 call eax