Just Let It Flow

October 4, 2010

When Does GetMessage Return -1?

Filed under: Windows — adeyblue @ 4:44 am

  1. Introduction
  2. Humble Beginnings
  3. The Guts
  4. A Quick test
  5. Wrap Up
  6. Appendix: Other Interesting Behaviour


Recently WinDevs, the guys in charge of developing the Getting Started content for native developers on MSDN, and I had a short conversation relating to message loops. They’re in the process of creating a Win32 tutorial in the vein of TheForgers teaching the basics of creating a small Direct2D drawing app. The message loop used in their tutorials is the ubiquitous:

MSG msg;
while(GetMessage(&msg, NULL, 0, 0))

On a forum I visit, one of the denizens had pointed out in response to an unrelated query that the GetMessage documentation states -1 can be returned on failure and that the loop above should not be used, instead advising programmers to use:

MSG msg;
BOOL bRet;
while( (bRet = GetMessage( &msg, NULL, 0, 0 )) != 0)
    if (bRet == -1)
        // handle the error and possibly exit

As the tutorial is aimed at beginners and no doubt intended to teach best practices, I mentioned the contradictory advice to them. After a bit of investigation, they’ve since explained the incongruity. MSDN mentions as examples that an invalid HWND and an invalid message pointer can be causes of a -1 return. Being curious about such things, I wondered what exactly causes such a return.

Humble Beginnings

In the grand scheme of things, GetMessage is a simple API. Firstly, it validates some input:

; GetMessageW, user32.dll, Win7
mov     edx, [ebp+wMsgFilterMin]
mov     ecx, [ebp+wMsgFilterMax]
or      edx, ecx ; or together both passed in filter values
mov     eax, 0FFFE0000h
test    edx, eax ; and them with 0xFFFE0000 to check if any of the high 15 bits are set
jnz     loc_77D4CF3A ; if any are, go to the following section
; ---
; loc_77D4CF3A
; ---
cmp     ecx, 0FFFFFFFFh ; Compare the max filter with UINT_MAX
jnz     short loc_77D4CF4B ; If it's not that, go to the location below
test    [ebp+wMsgFilterMin], eax
jnz     short loc_77D4CF4B ; if the min filter value isn't 0xFFFE0000, go to below location
xor     ecx, ecx  ; otherwise set max filter to 0
jmp     loc_77D28FB1 ; and go call into the kernel
; ---
; loc_77D4CF4B
; ---
push    57h
call    _UserSetLastError@4 ; SetLastError(Error_Invalid_parameter)
xor     eax, eax ; set return value as false
jmp     loc_77D28FDE ; goto following code
; ---
; loc_77D28FDE
; ---
pop     ebp
retn    10h ; return from function

Straightaway, we can see that passing an invalid message filter range won’t cause a -1 return, so that’s one off the list. After this validatation, the next upheaval is the transition to kernel mode, via a call to NtUserGetMessage.

The Guts

Upon entry to NtUserGetMessage, the message filters are validated again in the same way as above (because the function can be called direct in kernel mode) before a flying visit to xxxInternalGetMessage dumps us into xxxRealInternalGetMessage which is the workhorse for both Peek and GetMessage.

Three functions on from where we started, and the HWND value is finally examined:

; xxxRealInternalGetMessage, win32k.sys
push    24h
push    offset unk_BF9FF400
call    __SEH_prolog4 ; Setup SEH, we'll see this later
mov     esi, _gptiCurrentThreadInfo
mov     ebx, [ebp+hwnd]
cmp     ebx, 0FFFFFFFFh ; if the passed in HWND is (HWND)-1...
jz      short loc_BF8B95F6
cmp     ebx, 0FFFFh
jnz     short loc_BF8B95f9 ; or (HWND)0xFFFF...
; ---
; loc_BF8B95F6
; ---
xor     ebx, ebx
inc     ebx ; set the HWND to 1
; ---
; loc_BF8B95F9
; ---
test    ebx, ebx
jz      short loc_BF8B9645 ;  if HWND is NULL, go do thread message work
cmp     ebx, 1
jz      short loc_BF8B9645 ; likewise if it's 1
mov     ecx, ebx
call    @ValidateHwnd@4 ; Otherwise check it's a valid window
mov     edi, eax 
mov     [ebp+var_1C], edi ; Save the HWND's associated PWND struct on the stack
test    edi, edi
jnz     short loc_BF8B9626 ; check the PWND is not NULL. If it isn't, do window work
mov     eax, [ebp+pMsg] 
and     [eax], edi ; otherwise, set MSG.hwnd to NULL
and     [eax+4], edi ; and MSG.message to WM_NULL
; now here's the clever part. PeekMessage returns 0 on invalid HWND
; but GetMessage returns -1, this does both depending on the value of fromGetMessage
mov     eax, [ebp+fromGetMessage] 
neg     eax
sbb     eax, eax
jmp     loc_BF8B9B78 ; go to exit
; ---
; loc_BF8B9B78
; ---
call    __SEH_epilog4 ; teardown the SEH frame
retn    18h ; return and pop the stack

The documentation turns out to be correct and we have a -1 return when passed an invalid HWND. The other special window constants of -1 and 0xFFFF are documented values for PeekMessage to take but by virtue of the shared code are valid for GetMessage too.

As there’s an active SEH frame, let’s see what happens when an exception happens:

; ---
; loc_BF8B9B5E, the exception filter
; This is an equivalent of
; __except(__W32ExceptionHandler(GetExceptionCode()))
; ---
mov     eax, [ebp-14h]
mov     eax, [eax]
push    dword ptr [eax]
call    __W32ExceptionHandler@4 ; always returns EXCEPTION_EXECUTE_HANDLER
retn ; which leads into...
; ---
; loc_BF8B9B6B, the exception handler body
; ---
mov     esp, [ebp-18h] ; reset stack pointer to the functions stack space
mov     dword ptr [ebp-4], 0FFFFFFFEh
or      eax, 0FFFFFFFFh ; set return value to -1
; ---
; loc_BF8B9B78
; ---
call    __SEH_epilog4 ; teardown the SEH frame
retn    18h ; return -1

So, a second way for the function to return -1 is if an exception happens during its execution. This part only applies to Vista onwards, on XP there is no such exception handler set up in the function. If one occurs, a handler created by NtUserGetMessage will be entered, returning 0.

For non-exceptional cases other than the window validation, there‚Äôs only one exit route out of the function. In the second half, the ebx register contains the return value. It’s only assigned to once, after a message has been picked:

cmp     [ebp+fromGetMessage], 0 ; if from PeekMessage
jz      short loc_BF8B99C5 ; set return = 1
cmp     dword ptr [edi+4], 12h ; otherwise if from GetMessage and msg == WM_QUIT
jz      loc_BF8B9A8E ; set return to 0, otherwise continue and set return to 1
; ---
; loc_BF8B99C5
; ---
xor     ebx, ebx
inc     ebx
; ---
; loc_BF8B9A8E
; ---
xor     ebx, ebx

As far as the return value goes, that’s that. On Windows 7, we have four exit routes with a possible three values and two occasions we can receive -1.

A Quick Test

Windows 7 represents only one of the OS’s MSDN currently supports and while you can look at assembly all day, it’s resource intensive and not exactly fun. The simplest way to find out what happens on the other OS’s is to just try it. So that’s what I did. Note 1 contains some code that performs the five failure tests over which the user has control. After running it on various VM’s I had laying around, here are the results. LE denotes the return from GetLastError.

98 SE, this test used GetMessageA instead of GetMessageW like the others:

High word flags returned 1, (LE = 0)
NULL Msg returned 1, (LE = 0)
Read-Only Msg returned 1, (LE = 0)
Invalid Msg returned 1, (LE = 0)
Invalid HWND returned ffffffff, (LE = 0)

NT4 SP2:

High word flags returned 0, (LE = 87)
NULL Msg returned 0, (LE = 0)
Read-Only Msg returned 0, (LE = 0)
Invalid Msg returned 0, (LE = 0)
Invalid HWND returned ffffffff, (LE = 1400)


High word flags returned 0, (LE = 87)
NULL Msg EXCEPTION returned c0000005, (LE = 998)
Read-Only Msg returned 0, (LE = 998)
Invalid Msg EXCEPTION returned c0000005, (LE = 998)
Invalid HWND returned ffffffff, (LE = 1400)

7 SP0:

High word flags returned 0, (LE = 87)
NULL Msg EXCEPTION returned c0000005, (LE = 998)
Read-Only Msg returned 0, (LE = 998)
Invalid Msg EXCEPTION returned c0000005, (LE = 998)
Invalid HWND returned ffffffff, (LE = 1400)


A set of somewhat suprising results. It seems only NT4 had sane behaviour, with 98 not giving a hoot and returning success regardless. One explanation for this is that it returned whether there were messages present when given incorrect parameters, but that’s just a guess.

Slightly more worrying are the exceptions produced by XP and 7. This pours water on the claims that an invalid MSG pointer will return -1, when in such situations it may not return in the conventional sense at all. Let’s not forget common sense however, is anybody going to be calling a function whose sole operation is to fill in a struct and give it a bad pointer?

Wrap Up

Knowning what we know now, we can see that the return value section of the GetMessage doc page is correct, if only partially. Out of the 5 failure cases over which a user has control, only an invalid, non-null HWND actually returns -1. An invalid message pointer never does, returning 0 on older NT and 1 on 9x Windows, but worse, causing an access violation on new when given a completely wild pointer.

One full circle later and where have we ended up? Barring internal errors (which I’ve never encountered) GetMessage(&msg, NULL, 0, 0) will only return 0 upon receiving WM_QUIT with -1 an impossibility. In regards to Win devs and their message loops everwhere, it’s as you were.

Other Interesting Behaviour

Lost Messages

Something glossed over earlier in the transition between NtUserGetMessage and xxxGetMessage, is that a second MSG structure allocated on the kernel stack is actually filled in first rather than the pointer you give to GetMessage. After xxxGetMessage completes (successfully or not), its contents are copied to the original MSG structure. If something goes wrong during this copy, the message is lost. There are no second chances.

The Exceptions

XP and 7 cause exceptions when given a bad message because GetMessage doesn’t check the return value of NtUserGetMessage at all, nor the validity of the message pointer before dereferencing it.

push     esi
mov      esi,dword ptr [ebp+8] 
push     ecx  ; wFilterMax
push     dword ptr [ebp+10h] ; wFilterMin
push     dword ptr [ebp+0Ch] ; hWnd
push     esi ; pMsg
call     _NtUserGetMessage@16
mov      ecx,dword ptr [esi+4] ; access pMsg->message, if it's bad it goes boom
cmp      ecx,102h ; check if its WM_CHAR...
je       _GetMessageW@16+46h
cmp      ecx,0CCh ; or EM_SETPASSWORDCHAR
je       _GetMessageW@16+46h
pop      esi  
pop      ebp  
ret      10h

Note 1: The test program

// compile flags
// cl /GS- /DUNICODE /D_UNICODE /Ox test.cpp /link user32.lib kernel32.lib /subsystem:console /entry:entryPoint
#include <windows.h>
HANDLE g_hstdOut = NULL;
void PrintResult(LPCTSTR desc, BOOL ret)
   TCHAR buffer[150];
   int written = wsprintf(buffer, TEXT("%s returned %x, (LE = %d)\r\n"), desc, ret, GetLastError());
   DWORD wrote;
   WriteFile(g_hstdOut, buffer, written * sizeof(buffer[0]), &wrote, NULL);
#define DO_TEST(msg, hwnd, min, max, desc) \
   do \
   { \
       __try \
       { \
           SetLastError(0); \
           BOOL ret = GetMessage((msg), (hwnd), (min), (max)); \
           PrintResult((desc), ret); \
       } \
       { \
           PrintResult(desc TEXT(" EXCEPTION"), GetExceptionCode()); \
       } \
    } \
static const MSG readOnlyMessage = {0};
extern "C" int WINAPI entryPoint()
    g_hstdOut = GetStdHandle(STD_OUTPUT_HANDLE);
    MSG msg = {0};
    // create the message queue
    PeekMessage(&msg, NULL, 0, 0, PM_NOREMOVE);
    // make sure we have a message so GM won't block if it succeeds
    for(int i = 0; i < 10; ++i)
    	PostThreadMessage(GetCurrentThreadId(), WM_APP, 0, 0);
    // high word wake flags don't see to be taken into account pre Win2000
    DO_TEST(&msg, NULL, 0x12345678, 0xDCBA9876, TEXT("High word flags"));
    DO_TEST(NULL, NULL, 0, 0, TEXT("NULL Msg"));
    DO_TEST((PMSG)&readOnlyMessage, NULL, 0, 0, TEXT("Read-Only Msg"));
    DO_TEST((PMSG)3, NULL, 0, 0, TEXT("Invalid Msg"));
    DO_TEST(&msg, (HWND)0x9, 0, 0, TEXT("Invalid HWND"));

No Comments »

No comments yet.

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by WordPress