Just Let It Flow

December 16, 2014

Direct 2 D Scene of the Accident

Filed under: Code,Windows — adeyblue @ 10:45 pm

I don’t know much about Direct2D, heck I don’t know much about Direct3D. I’m not a graphics programmer, the only game I ever made was a Java version of Paratrooper. Regardless, its always useful to know more, so in order to have a little dabble and gentle easing in to this supposed successor of GDI, I thought I’d translate something that’s not too involved but still useful.

I’ve been messing with Pete’s Software renderer for ePSXe the PlayStation emulator so, since it uses DirectDraw, I decided to translate that. Now most of Microsoft’s newer native code API’s are targeted at C++, with bindings for C. It just so happens that this software renderer uses C, so no problem. I converted the initialization function, ran in it the debugger to make sure nothing went wrong and:

Direct2D - Run Time Check Error #0 - ESP Not Properly Saved

Direct2D – Run Time Check Error #0 – ESP Not Properly Saved

Oh, don’t think that was meant ot happen. Time for a standalone example, since ePSXe loads a lot of dlls.

#define WIN32_LEAN_AND_MEAN   
#define CINTERFACE   
#include <d2d1.h>  
#include <stdio.h>   
#pragma comment(lib, "d2d1.lib")   
int __cdecl main()   
    ID2D1Factory* pFactory = NULL;   
    HRESULT hr = S_OK;   
    D2D1_SIZE_U targetSize;   
    D2D1_SIZE_U pixSize;   
    HWND hwndCon = GetConsoleWindow();   
    RECT winSize;   
    ID2D1HwndRenderTarget* pHwndRenderTarget = NULL;   
    hr = D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, IID_ID2D1Factory, &opts, (void**)&pFactory);   
    GetClientRect(hwndCon, &winSize);   
    pixSize.width = winSize.right;   
    pixSize.height = winSize.bottom;   
    renderProps.type = D2D1_RENDER_TARGET_TYPE_DEFAULT;   
    renderProps.pixelFormat = pixFmt;   
    renderProps.minLevel = D2D1_FEATURE_LEVEL_DEFAULT;   
    renderProps.usage = D2D1_RENDER_TARGET_USAGE_NONE;   
    renderProps.dpiX = renderProps.dpiY = 0.f;   
    hwndRenderProps.hwnd = hwndCon;   
    hwndRenderProps.pixelSize = pixSize;   
    hwndRenderProps.presentOptions = D2D1_PRESENT_OPTIONS_NONE;   
    if(FAILED(hr = ID2D1Factory_CreateHwndRenderTarget(pFactory, &renderProps, &hwndRenderProps, &pHwndRenderTarget)))   
        printf("Failed to create render target, err = %#x\n", hr);   
        return 0;   
    pixFmt = ID2D1HwndRenderTarget_GetPixelFormat(pHwndRenderTarget);   
    printf("HWND RT PixFmt: %lu, alpha: %lu\n", pixFmt.format, pixFmt.alphaMode);   

Right, tried again and

HwndRenderTarget GetPixelFormat crash

HwndRenderTarget GetPixelFormat crash

Huh!? Ok. So there’s definitely some problem with GetPixelFormat. If you’ve programmed for any length of time, you’ll recognise the first error above as meaning ‘the function prototype has the wrong number of parameters or wrong calling convention’. The docs for the function disagree that this is the problem since it takes no parameters (beside the interface pointer which is implicit for C++), and the calling convention is definitely stdcall since all Microsoft interface-based api’s are. So what gives?

For some reason which I can’t remember, I coverted it to use the C++ interface to see if that’d change anything, although there’s no way it could.

How did that happen? It works!

How did that happen? It works!

What? Something crazy is going on here. How did that work? All that changed was that the first parameter was passed implicitly. Actually, that wasn’t true at all. Oh, it did happen, but so did something else.

To go back to the C compiler/interface[1] for a minute, this is the generated code. It’s nothing special, exactly what you’d expect for a COM interface call taking no parameters:

;    pixFmt = ID2D1HwndRenderTarget_GetPixelFormat(pHwndRenderTarget);   
00411384  mov         eax,dword ptr [ebp-6Ch]  ; interface ptr
00411387  push        eax                      ; pass interface as only parameter
00411388  mov         ecx,dword ptr [ebp-6Ch]  ; interface ptr
0041138B  mov         edx,dword ptr [ecx]      ; vtable
0041138D  mov         eax,dword ptr [edx+0C8h] ; function to call
00411393  call        eax                      ; call it

You’d expect the C++ code to look identical, and it mostly does. Except it also sneakily causes the function to allocate more stack space, and passes an extra argument to the function.

;    pixFmt = pHwndRenderTarget->GetPixelFormat();
00411384  lea         eax,[ebp-0B4h]           ; take address of something on the stack
0041138A  push        eax                      ; pass it as a parameter
0041138B  mov         ecx,dword ptr [ebp-6Ch]  ; interface ptr
0041138E  mov         edx,dword ptr [ecx]      ; vtable
00411390  mov         eax,dword ptr [ebp-6Ch]  ; interface ptr again
00411393  push        eax                      ; pass interface as parameter
00411394  mov         ecx,dword ptr [edx+0C8h] ; function to call
0041139A  call        ecx                      ; call it

Again, what’s going on here. Does GetPixelFormat take one or two parameters? Or maybe it’s actally a stdcall varags function that somehow cleans the stack itself? To the disassembly!

The implementation in question here[2] isn’t very long, but contains a lot of noise for our purposes. So I’ve trimmed out the bits we don’t need, the full code is below.

; D2DDCRenderTarget::GetPixelFormat:
729E9FF4  mov         esi,dword ptr [ebp+8] ; first parameter, so it definitely takes at least one
729E9FF7  mov         ecx,dword ptr [esi+8] ; It's a 12 byte structure at least
729E9FFA  mov         eax,dword ptr [ecx] ; One of it's members is a class
729E9FFC  push        edi  
729E9FFD  mov         dword ptr [ebp-14h],ecx 
729EA000  call        dword ptr [eax+0Ch] ; Since it calls a thiscall function
729EA045  mov         edi,dword ptr [ebp+0Ch] ; A-ha, a second argument so it takes at least two
729EA048  mov         ecx,eax               ; get some-
729EA04A  mov         eax,dword ptr [ecx]   ; data into-
729EA04C  mov         ecx,dword ptr [ecx+4] ; the right registers
729EA04F  mov         dword ptr [edi],eax   ; We already knew the argument was a pointer, but here's proof it's at least a DWORD
729EA051  mov         dword ptr [edi+4],ecx ; And here's proof it's at least 2 dwords, so 8 bytes
729EA064  mov         eax,edi ; edi has been unchanged since above so it returns the second argument
729EA066  pop         edi
729EA067  pop         esi
729EA068  leave
729EA069  ret         8 ; and here's final proof it takes two arguments

That seems pretty conclusive. It takes two parameters, the interface pointer and another pointer that it also returns. Since we know it returns a D2D1_PIXEL_FORMAT structure, and those are composed of two DWORD sized values, that’s what the second argument is a pointer to.

In short, the C interface to Direct2D is broken. It doesn’t account for the automagic argument passing of the C++ compiler which the definitions seem to rely on. It’s not specific to GetPixelFormat either, that’s just where I noticed it. All of the D2D functions that return structures seem to be affected, this includes render target functions like GetSize and GetPixelSize, and brush functions like ID2D1SolidColorBrush::GetColor.

Talking of ID2D1SolidColorBrush::GetColor, the compiler weirdness doesn’t end there. This function is like a red rag to a bull and causes the compiler to get stuck between both worlds. With the C interface, calling it produces some really wacky code:

;	black = ID2D1SolidColorBrush_GetColor(pBrush);
00411409  mov         eax,dword ptr [ebp-74h] ; Interface ptr
0041140C  push        eax                     ; Pass it as the 2nd parameter!
0041140D  mov         ecx,dword ptr [ebp-74h] ; Interface ptr
00411410  mov         edx,dword ptr [ecx]     ; Vtable
00411412  lea         eax,[ebp-0F0h]          ; Invent a stack parameter
00411418  push        eax                     ; Pass it as the first parameter!
00411419  mov         ecx,dword ptr [edx+24h] ; Function to call
0041141C  call        ecx                     ; call it

Even though it got the number of parameters right, the call would probably still crash since the parameters are passed in the wrong order. I’m not sure how you make code that messes up the compiler this badly, but Direct2D managed it.

All these problems are fixable, but require you to do that questionable practice of editing the headers. All that needs to be done is adding a pointer to the struct they return as the last parameter. The return value also needs updating to a pointer otherwise assigning the return value will produce incorrect values.

Not the greatest introduction I’ve ever had to an API.


[1]: The disassembly in this post was produced by Visual Studio 2013, I tried all the code and variations in VS2008 as well and the relevant output was identical so either this isn’t a compiler bug, or a longstanding one.

[2]: There are other implementations of GetPixelFormat for different render target types, they all behave the same in regards to this post, MSDN’s definitions and the Direct2D headers.


729E9FEB  mov         edi,edi 
729E9FED  push        ebp  
729E9FEE  mov         ebp,esp 
729E9FF0  sub         esp,14h 
729E9FF3  push        esi  
729E9FF4  mov         esi,dword ptr [ebp+8] 
729E9FF7  mov         ecx,dword ptr [esi+8] 
729E9FFA  mov         eax,dword ptr [ecx] 
729E9FFC  push        edi  
729E9FFD  mov         dword ptr [ebp-14h],ecx 
729EA000  call        dword ptr [eax+0Ch] 
729EA003  lea         ecx,[ebp-10h] 
729EA006  call        FPUStateX87<0>::FPUStateX87<0> (729DDFF1h) 
729EA00B  lea         eax,[ebp-0Ch] 
729EA00E  call        FPUStateSSE<1>::FPUStateSSE<1> (729DE039h) 
729EA013  cmp         dword ptr [esi+1A0h],0 
729EA01A  jne         D2DDCRenderTarget::GetPixelFormat+4Dh (729EA038h) 
729EA01C  cmp         dword ptr [esi+0C0h],0 
729EA023  jne         D2DDCRenderTarget::GetPixelFormat+4Dh (729EA038h) 
729EA025  mov         ecx,dword ptr [esi+58h] 
729EA028  mov         eax,dword ptr [ecx+40h] 
729EA02B  mov         edi,dword ptr [ebp+0Ch] 
729EA02E  mov         dword ptr [edi],eax 
729EA030  mov         eax,dword ptr [ecx+44h] 
729EA033  mov         dword ptr [edi+4],eax 
729EA036  jmp         D2DDCRenderTarget::GetPixelFormat+69h (729EA054h) 
729EA038  lea         eax,[ebp-8] 
729EA03B  push        eax  
729EA03C  lea         eax,[esi+10h] 
729EA03F  push        eax  
729EA040  call        DrawingContext::GetPixelFormat (729F8512h) 
729EA045  mov         edi,dword ptr [ebp+0Ch] 
729EA048  mov         ecx,eax 
729EA04A  mov         eax,dword ptr [ecx] 
729EA04C  mov         ecx,dword ptr [ecx+4] 
729EA04F  mov         dword ptr [edi],eax 
729EA051  mov         dword ptr [edi+4],ecx 
729EA054  lea         esi,[ebp-10h] 
729EA057  call        CFloatFPU::~CFloatFPU (729DDFC5h) 
729EA05C  mov         ecx,dword ptr [ebp-14h] 
729EA05F  mov         edx,dword ptr [ecx] 
729EA061  call        dword ptr [edx+10h] 
729EA064  mov         eax,edi 
729EA066  pop         edi  
729EA067  pop         esi  
729EA068  leave            
729EA069  ret         8

I looked at the disassembly and this is produced for C, the [url=http://msdn.microsoft.com/en-us/library/windows/desktop/dd316814(v=vs.85).aspx]function is declared as taking no parameters[/url], so just the interface is pushed
pixFmt = ID2D1HwndRenderTarget_GetPixelFormat(pHwndRenderTarget);
00411384 mov eax,dword ptr [ebp-6Ch]
00411387 push eax
00411388 mov ecx,dword ptr [ebp-6Ch]
0041138B mov edx,dword ptr [ecx]
0041138D mov eax,dword ptr [edx+0C8h]
00411393 call eax

Powered by WordPress