It appears to be a race condition because this call was done after casablanca110.dll is unloaded. I was trying to detect memory leak using umdh command so I simply added a breakpoint to FreeLibrary. I was running my unit test using SDK's unittestpp's TestRunner.exe. So on the last FreeLibrary where TestRunner says everything is done (a bit after that) I reached this exception. I loaded back the DLL to see which API it was. If I run the test without breakpoint there is no exception. All my modules are unloaded already (no stack touching my own code)
FAULTING_IP:
casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ)+0
000007fe`d58a786a ?? ???
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007fed58a786a (casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ))
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000008
Parameter[1]: 000007fed58a786a
Attempt to execute non-executable address 000007fed58a786a
FAULTING_THREAD: 0000000000000c90
PROCESS_NAME: TestRunner.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 0000000000000008
EXCEPTION_PARAMETER2: 000007fed58a786a
WRITE_ADDRESS: 000007fed58a786a
FOLLOWUP_IP:
casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ)+0
000007fe`d58a786a ?? ???
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 92300000
APPLICATION_VERIFIER_FLAGS: 0
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_SOFTWARE_NX_FAULT_CODE
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ_CODE
DEFAULT_BUCKET_ID: INVALID_POINTER_READ_CODE
LAST_CONTROL_TRANSFER: from 000007fef93a49fb to 000007fed58a786a
STACK_TEXT:
00000000`0527f798 000007fe`f93a49fb : 00000000`05bc0c80 00000000`05bc0c80 00000000`01a00000 00000000`00000007 : casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ)
00000000`0527f7a0 000007fe`f93a48eb : 00000000`00000000 00000000`0628ff10 00000000`00000000 00000000`06021f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_SafeAppCallback+0x14f
00000000`0527f830 000007fe`f93a488c : 00000000`00000000 00000000`05e65cc0 00000000`00000000 00000000`06021f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_ControlledAppCallback+0x5d
00000000`0527f860 000007fe`f9396ec7 : 00000000`00000000 00000000`00000007 00000000`026fdf70 00000000`76f6699f : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::IndicateCompletionStatusCommon+0x6e
00000000`0527f8a0 000007fe`f93adbc3 : 00000000`06042f50 00000000`00000007 00000000`00000000 000007fe`f93914c7 : WINHTTP!HTTP_USER_REQUEST::_IndicateIoCompletion+0x12e
00000000`0527f8f0 000007fe`f93adafb : 00000000`06042f50 00000000`00000007 00000000`00000000 00000000`00000007 : WINHTTP!HTTP_USER_REQUEST::_OnAsyncQueryDataAvailable+0xc7
00000000`0527f920 000007fe`f9397012 : 00000000`00000000 00000000`05d9bea8 00000000`00000000 00000000`0527fb98 : WINHTTP!HTTP_USER_REQUEST::OnReadData+0x82
00000000`0527f970 000007fe`f9391b40 : 00000000`00000007 00000000`00000000 80000000`00000000 00000000`029d2fc0 : WINHTTP!WEBIO_REQUEST::OnIoComplete+0x184
00000000`0527f9d0 00000000`76f62484 : 00000000`029d2f10 00000000`029d2f10 00000000`00000000 00000000`029d2fc0 : WINHTTP!HTTP_THREAD_POOL::_WorkItemCallback+0xfd
00000000`0527fa30 00000000`76f66bd2 : 00000000`770545e8 00000000`06132f98 00000000`770545c0 00000000`77054610 : ntdll!TppWorkpExecuteCallback+0xa4
00000000`0527fa90 00000000`76d4652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x6c9
00000000`0527fd90 00000000`76f7c521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0527fdc0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: casablanca110!ILT+26725
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: casablanca110
IMAGE_NAME: casablanca110.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 51930804
STACK_COMMAND: ~15s ; kb
FAILURE_BUCKET_ID: INVALID_POINTER_READ_CODE_c0000005_casablanca110.dll!ILT
BUCKET_ID: X64_APPLICATION_FAULT_INVALID_POINTER_READ_SOFTWARE_NX_FAULT_CODE_casablanca110!ILT+26725
Followup: MachineOwner
---------
Comments: I think this is result of my code running code in a different threads and when the unittestpp finishes it will unload test modules (including casablanca dll) but things have not completely terminated (or cancelled) in time. So I thought it's a matter of timing so I added 5s wait and it was still crashing (always at the same place now) but none of my code in stacks and my test DLL is unloaded. All my code does is do a request (which is long lived and listens for response in a loop) and stop everything (basically open/close connection type of test to catch this issue; when I use /loop:15 or more it runs into this issue). I suspect regular quick google.com query would not run in this type of issue. 000007fe`d6dbf1ea : 00000000`08e7efe0 cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : casablanca110!std::_Ref_count_base::_Incref+0xf 000007fe`d6d8a5de : 00000000`0f9be5f0 00000000`08e7eff0 00000000`08e7efe0 cccccccc`cccccccc : casablanca110!std::_Ptr_base<pplx::scheduler_interface>::_Reset+0x3a 000007fe`d6d9df9c : 00000000`0f9be5f0 000007fe`d6fcdb38 cccccccc`cccccccc cccccccc`cccccccc : casablanca110!std::_Ptr_base<pplx::scheduler_interface>::_Reset<pplx::scheduler_interface>+0x3e 000007fe`d6e90d69 : 00000000`0f9be5f0 000007fe`d6fcdb38 cccccccc`cccccccc cccccccc`cccccccc : casablanca110!std::shared_ptr<pplx::scheduler_interface>::shared_ptr<pplx::scheduler_interface>+0x3c 000007fe`d6da139b : 00000000`0f9be5f0 cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : casablanca110!pplx::get_ambient_scheduler+0xa9 000007fe`d6dc83d4 : 00000000`0f9be9e8 00000000`00000007 00000000`0f9be668 cccccccc`cccccccc : casablanca110!pplx::task_options::task_options+0x3b 000007fe`f93a49fb : 00000000`06ed4c80 00000000`066b3f10 00000000`00080000 00000000`13d62ff0 : casablanca110!web::http::client::details::winhttp_client::completion_callback+0xb24 000007fe`f93a48eb : 00000000`00000001 000007fe`d6d67b94 00000000`00000001 00000000`13ba8f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_SafeAppCallback+0x14f 000007fe`f93a488c : 00000000`00000000 00000000`13ba8f80 00000000`06ed4c80 00000000`13ba8f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_ControlledAppCallback+0x5d 000007fe`f93b4be3 : 00000000`00000000 00000000`00000007 00000000`06ed4c80 00000000`00000000 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::IndicateCompletionStatusCommon+0x6e 000007fe`d6dc8278 : 00000000`066b3fd8 00000000`13d62ff0 00000000`00000007 cccccccc`cccccccc : WINHTTP!WinHttpReadData+0x33d 000007fe`f93a49fb : 00000000`06ed4c80 00000000`066b3f10 00000000`00040000 00000000`13a99fa4 : casablanca110!web::http::client::details::winhttp_client::completion_callback+0x9c8 000007fe`f93a48eb : 00000000`00000000 000007fe`ee9190f5 00000000`00000000 00000000`13a99f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_SafeAppCallback+0x14f 000007fe`f93a488c : 00000000`00000000 00000000`09513cc0 00000000`00000000 00000000`13a99f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_ControlledAppCallback+0x5d 000007fe`f9396ec7 : 00000000`00000000 00000000`00000000 00000000`00000000 000007fe`ee91a32d : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::IndicateCompletionStatusCommon+0x6e 000007fe`f93adbc3 : 00000000`136fcf50 00000000`00000007 00000000`00000000 000007fe`f93914c7 : WINHTTP!HTTP_USER_REQUEST::_IndicateIoCompletion+0x12e 000007fe`f93adafb : 00000000`136fcf50 00000000`00000007 00000000`00000000 000007fe`00000007 : WINHTTP!HTTP_USER_REQUEST::_OnAsyncQueryDataAvailable+0xc7 000007fe`f9397012 : 00000000`00000000 00000000`0938cea8 00000000`00000000 00000000`0f9bf7d8 : WINHTTP!HTTP_USER_REQUEST::OnReadData+0x82 000007fe`f9391b40 : 00000000`00000007 00000000`00000000 80000000`00000000 000007fe`f9391a64 : WINHTTP!WEBIO_REQUEST::OnIoComplete+0x184 000007fe`ee920009 : 00000000`04383eb0 00000000`04383eb0 00000000`00000000 00000000`0f9bf7d8 : WINHTTP!HTTP_THREAD_POOL::_WorkItemCallback+0xfd 00000000`76f62484 : 00000000`08f5cf10 00000000`08f5cfc0 00000000`0f9bf7d8 00000000`08f5cfc0 : vfbasics!AVrfpTpWorkCallback+0x51 00000000`76f70c26 : 00000000`770545e8 00000000`0925ff98 00000000`770545c0 00000000`77054610 : ntdll!TppWorkpExecuteCallback+0xa4 00000000`76d4652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x5ff 00000000`76f7c521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0xd 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d
FAULTING_IP:
casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ)+0
000007fe`d58a786a ?? ???
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007fed58a786a (casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ))
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000008
Parameter[1]: 000007fed58a786a
Attempt to execute non-executable address 000007fed58a786a
FAULTING_THREAD: 0000000000000c90
PROCESS_NAME: TestRunner.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 0000000000000008
EXCEPTION_PARAMETER2: 000007fed58a786a
WRITE_ADDRESS: 000007fed58a786a
FOLLOWUP_IP:
casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ)+0
000007fe`d58a786a ?? ???
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 92300000
APPLICATION_VERIFIER_FLAGS: 0
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_SOFTWARE_NX_FAULT_CODE
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ_CODE
DEFAULT_BUCKET_ID: INVALID_POINTER_READ_CODE
LAST_CONTROL_TRANSFER: from 000007fef93a49fb to 000007fed58a786a
STACK_TEXT:
00000000`0527f798 000007fe`f93a49fb : 00000000`05bc0c80 00000000`05bc0c80 00000000`01a00000 00000000`00000007 : casablanca110!ILT+26725(?completion_callbackwinhttp_clientdetailsclienthttpwebCAXPEAX_KK0KZ)
00000000`0527f7a0 000007fe`f93a48eb : 00000000`00000000 00000000`0628ff10 00000000`00000000 00000000`06021f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_SafeAppCallback+0x14f
00000000`0527f830 000007fe`f93a488c : 00000000`00000000 00000000`05e65cc0 00000000`00000000 00000000`06021f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_ControlledAppCallback+0x5d
00000000`0527f860 000007fe`f9396ec7 : 00000000`00000000 00000000`00000007 00000000`026fdf70 00000000`76f6699f : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::IndicateCompletionStatusCommon+0x6e
00000000`0527f8a0 000007fe`f93adbc3 : 00000000`06042f50 00000000`00000007 00000000`00000000 000007fe`f93914c7 : WINHTTP!HTTP_USER_REQUEST::_IndicateIoCompletion+0x12e
00000000`0527f8f0 000007fe`f93adafb : 00000000`06042f50 00000000`00000007 00000000`00000000 00000000`00000007 : WINHTTP!HTTP_USER_REQUEST::_OnAsyncQueryDataAvailable+0xc7
00000000`0527f920 000007fe`f9397012 : 00000000`00000000 00000000`05d9bea8 00000000`00000000 00000000`0527fb98 : WINHTTP!HTTP_USER_REQUEST::OnReadData+0x82
00000000`0527f970 000007fe`f9391b40 : 00000000`00000007 00000000`00000000 80000000`00000000 00000000`029d2fc0 : WINHTTP!WEBIO_REQUEST::OnIoComplete+0x184
00000000`0527f9d0 00000000`76f62484 : 00000000`029d2f10 00000000`029d2f10 00000000`00000000 00000000`029d2fc0 : WINHTTP!HTTP_THREAD_POOL::_WorkItemCallback+0xfd
00000000`0527fa30 00000000`76f66bd2 : 00000000`770545e8 00000000`06132f98 00000000`770545c0 00000000`77054610 : ntdll!TppWorkpExecuteCallback+0xa4
00000000`0527fa90 00000000`76d4652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x6c9
00000000`0527fd90 00000000`76f7c521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`0527fdc0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: casablanca110!ILT+26725
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: casablanca110
IMAGE_NAME: casablanca110.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 51930804
STACK_COMMAND: ~15s ; kb
FAILURE_BUCKET_ID: INVALID_POINTER_READ_CODE_c0000005_casablanca110.dll!ILT
BUCKET_ID: X64_APPLICATION_FAULT_INVALID_POINTER_READ_SOFTWARE_NX_FAULT_CODE_casablanca110!ILT+26725
Followup: MachineOwner
---------
Comments: I think this is result of my code running code in a different threads and when the unittestpp finishes it will unload test modules (including casablanca dll) but things have not completely terminated (or cancelled) in time. So I thought it's a matter of timing so I added 5s wait and it was still crashing (always at the same place now) but none of my code in stacks and my test DLL is unloaded. All my code does is do a request (which is long lived and listens for response in a loop) and stop everything (basically open/close connection type of test to catch this issue; when I use /loop:15 or more it runs into this issue). I suspect regular quick google.com query would not run in this type of issue. 000007fe`d6dbf1ea : 00000000`08e7efe0 cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : casablanca110!std::_Ref_count_base::_Incref+0xf 000007fe`d6d8a5de : 00000000`0f9be5f0 00000000`08e7eff0 00000000`08e7efe0 cccccccc`cccccccc : casablanca110!std::_Ptr_base<pplx::scheduler_interface>::_Reset+0x3a 000007fe`d6d9df9c : 00000000`0f9be5f0 000007fe`d6fcdb38 cccccccc`cccccccc cccccccc`cccccccc : casablanca110!std::_Ptr_base<pplx::scheduler_interface>::_Reset<pplx::scheduler_interface>+0x3e 000007fe`d6e90d69 : 00000000`0f9be5f0 000007fe`d6fcdb38 cccccccc`cccccccc cccccccc`cccccccc : casablanca110!std::shared_ptr<pplx::scheduler_interface>::shared_ptr<pplx::scheduler_interface>+0x3c 000007fe`d6da139b : 00000000`0f9be5f0 cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : casablanca110!pplx::get_ambient_scheduler+0xa9 000007fe`d6dc83d4 : 00000000`0f9be9e8 00000000`00000007 00000000`0f9be668 cccccccc`cccccccc : casablanca110!pplx::task_options::task_options+0x3b 000007fe`f93a49fb : 00000000`06ed4c80 00000000`066b3f10 00000000`00080000 00000000`13d62ff0 : casablanca110!web::http::client::details::winhttp_client::completion_callback+0xb24 000007fe`f93a48eb : 00000000`00000001 000007fe`d6d67b94 00000000`00000001 00000000`13ba8f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_SafeAppCallback+0x14f 000007fe`f93a488c : 00000000`00000000 00000000`13ba8f80 00000000`06ed4c80 00000000`13ba8f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_ControlledAppCallback+0x5d 000007fe`f93b4be3 : 00000000`00000000 00000000`00000007 00000000`06ed4c80 00000000`00000000 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::IndicateCompletionStatusCommon+0x6e 000007fe`d6dc8278 : 00000000`066b3fd8 00000000`13d62ff0 00000000`00000007 cccccccc`cccccccc : WINHTTP!WinHttpReadData+0x33d 000007fe`f93a49fb : 00000000`06ed4c80 00000000`066b3f10 00000000`00040000 00000000`13a99fa4 : casablanca110!web::http::client::details::winhttp_client::completion_callback+0x9c8 000007fe`f93a48eb : 00000000`00000000 000007fe`ee9190f5 00000000`00000000 00000000`13a99f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_SafeAppCallback+0x14f 000007fe`f93a488c : 00000000`00000000 00000000`09513cc0 00000000`00000000 00000000`13a99f80 : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::_ControlledAppCallback+0x5d 000007fe`f9396ec7 : 00000000`00000000 00000000`00000000 00000000`00000000 000007fe`ee91a32d : WINHTTP!HTTP_REQUEST_HANDLE_OBJECT::IndicateCompletionStatusCommon+0x6e 000007fe`f93adbc3 : 00000000`136fcf50 00000000`00000007 00000000`00000000 000007fe`f93914c7 : WINHTTP!HTTP_USER_REQUEST::_IndicateIoCompletion+0x12e 000007fe`f93adafb : 00000000`136fcf50 00000000`00000007 00000000`00000000 000007fe`00000007 : WINHTTP!HTTP_USER_REQUEST::_OnAsyncQueryDataAvailable+0xc7 000007fe`f9397012 : 00000000`00000000 00000000`0938cea8 00000000`00000000 00000000`0f9bf7d8 : WINHTTP!HTTP_USER_REQUEST::OnReadData+0x82 000007fe`f9391b40 : 00000000`00000007 00000000`00000000 80000000`00000000 000007fe`f9391a64 : WINHTTP!WEBIO_REQUEST::OnIoComplete+0x184 000007fe`ee920009 : 00000000`04383eb0 00000000`04383eb0 00000000`00000000 00000000`0f9bf7d8 : WINHTTP!HTTP_THREAD_POOL::_WorkItemCallback+0xfd 00000000`76f62484 : 00000000`08f5cf10 00000000`08f5cfc0 00000000`0f9bf7d8 00000000`08f5cfc0 : vfbasics!AVrfpTpWorkCallback+0x51 00000000`76f70c26 : 00000000`770545e8 00000000`0925ff98 00000000`770545c0 00000000`77054610 : ntdll!TppWorkpExecuteCallback+0xa4 00000000`76d4652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x5ff 00000000`76f7c521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0xd 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d