Just Let It Flow

February 25, 2009

When Returning TRUE Returned FALSE

Filed under: Windows,Wow, that was stupid — adeyblue @ 1:49 am

The Dbghelp Symbol API, Deferred Loads and ERROR_CANCELLED (1223)

It was a day like any other, the birds were singing, bees were trying to have sex with them (as is my understanding) and my code worked was in a working state, but not for much longer. The symbol handling part of the code was in need of a cleanup so I took the opportunity to change the symbol loading options so that the api used the more efficient (apparently) deferred loading instead of buliding the full symbol table on initialization. That’s when the pain started.

Instead of the tests working fine, SymLoadModuleExW was returning failure for every single module passed to it. As I was investigating Process Explorer at the time, I hooked up the debugger to double check I was passing the same parameters it was, check; affirm the callback (below) was returning TRUE, double check; turned off deferred loading to see if that solved it, triple check.

BOOL CALLBACK DbgCallbackProc(HANDLE hProcess, ULONG ActionCode,
                              ULONG64 CallbackData, ULONG64 UserContext)
{
    BOOL bRet = TRUE;
    switch(ActionCode)
    {
        case CBA_READ_MEMORY:
        {
            IMAGEHLP_CBA_READ_MEMORY* pMem = (IMAGEHLP_CBA_READ_MEMORY*)CallbackData;
            bRet = ReadProcessMemory(hProcess, pMem->addr,
                         pMem->buf, pMem->bytes, pMem->bytesread);
        }
        break;
    }
    UNREFERENCED_PARAMETER(UserContext);
    return bRet;
}

Argh, it must’ve been somewhere inside SymLoadModuleExW that was failing, so I went in with the debugger. It did it’s setting up, it called the callback which returned TRUE, and somehow from this it synthesized ERROR_CANCELLED. It seemed that unlike practically every other Winapi callback, returning TRUE as a default didn’t mean “Yes, go away and do it” but exactly the opposite. That’s when I actually looked at the name at the name of the deferred loading action. CBA_DEFERRED_SYMBOL_LOAD_CANCEL, well, it doesn’t take a genius to figure it out, but having one on hand would’ve saved a whole bunch of time.

I changed the callback to:

BOOL CALLBACK DbgCallbackProc(HANDLE hProcess, ULONG ActionCode,
                              ULONG64 CallbackData, ULONG64 UserContext)
{
    BOOL bRet = TRUE;
    switch(ActionCode)
    {
        case CBA_READ_MEMORY:
        {
            IMAGEHLP_CBA_READ_MEMORY* pMem = (IMAGEHLP_CBA_READ_MEMORY*)CallbackData;
            bRet = ReadProcessMemory(hProcess, pMem->addr,
                         pMem->buf, pMem->bytes, pMem->bytesread);
        }
        break;
        case CBA_DEFERRED_SYMBOL_LOAD_CANCEL:
        {
            bRet = FALSE;
        }
        break;
    }
    UNREFERENCED_PARAMETER(UserContext);
    return bRet;
}

And everything returned to its original peaceful and tranquil state.

No Comments »

No comments yet.

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by WordPress