Forums - Adreno Vulkan drivers crash when using GWP-Asan

GWP-Asan (https://developer.android.com/ndk/guides/gwp-asan) is an Android 11 feature that helps catch memory errors. It picks a random subset of memory allocations and checks them for use-after-free and overflow errors. We have enabled it for the store build of our games.

We have observed crashes coming through to the Google Play Console in qglinternal::vkResetCommandPool on a number of devices:

- OnePlus 8T
- OnePlus 8
- OnePlus 8 Pro
- Samsung Galaxy S20 5G
- Samsung Galaxy Note20 Ultra 5G
- Redmi Note 9S

I got hold of a OnePlus 8T to try to repro the problem which indicated the crash is triggered by GWP-Asan detecting a use-after-free in the driver code. Here's the log of the crash with company information redacted:

    --------- beginning of crash
2020-12-03 15:31:47.074 21928-5546/? A/libc: Fatal signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x7bd67eddf4 in tid 5546 (VkThread 3260), pid 21928 (mycompany.mygame)
2020-12-03 15:31:47.151 6161-6161/? I/crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstone
2020-12-03 15:31:47.151 801-801/? I/tombstoned: received crash request for pid 5546
2020-12-03 15:31:47.152 6161-6161/? I/crash_dump64: performing dump of process 21928 (target tid = 5546)
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: Build fingerprint: 'OnePlus/OnePlus8T_EEA/OnePlus8T:11/RP1A.201005.001/2011132154:user/release-keys'
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: Revision: '0'
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: ABI: 'arm64'
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: Timestamp: 2020-12-03 15:31:47+0000
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: pid: 21928, tid: 5546, name: VkThread 3260  >>> com.mycompany.mygame <<<
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: uid: 10248
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x7bd67eddf4
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG: Cause: [GWP-ASan]: Use After Free, 524 bytes into a 1048-byte allocation at 0x7bd67edbe8
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x0  0000007ed03a7164  x1  0000007ed03a7164  x2  0000007b2101a458  x3  0000000000000020
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x4  0000000000000010  x5  0000000000000004  x6  0000000000001770  x7  0000000000000001
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x8  0000000000000000  x9  0000000000000000  x10 b400007bd67edbe8  x11 0000000000000030
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x12 0000000000000000  x13 0000000000000000  x14 0000000000000001  x15 00000000000003e0
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x16 0000007b211cea90  x17 0000007ed0314ad4  x18 0000007b0a99c000  x19 b400007cdf911820
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x20 0000000000000000  x21 0000000000000001  x22 0000007b2102dd14  x23 0000000000000000
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x24 000000000000000a  x25 0000000000000001  x26 0000007b5c690640  x27 0000000000000000
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     x28 0000007b5cb05a40  x29 0000007b2101a490
2020-12-03 15:31:47.160 6161-6161/? A/DEBUG:     lr  0000007b210dd044  sp  0000007b2101a490  pc  0000007b210dd050  pst 0000000060001000
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG: backtrace:
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #00 pc 00000000000be050  /vendor/lib64/hw/vulkan.adreno.so (BuildId: e52a46a3b09769b61e4e69576af2656e)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #01 pc 000000000008e15c  /vendor/lib64/hw/vulkan.adreno.so (BuildId: e52a46a3b09769b61e4e69576af2656e)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #02 pc 000000000008a560  /vendor/lib64/hw/vulkan.adreno.so (qglinternal::vkResetCommandPool(VkDevice_T*, VkCommandPool_T*, unsigned int)+40) (BuildId: e52a46a3b09769b61e4e69576af2656e)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #03 pc 000000000065d908  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanCommandBuffers::ResetCommandPool(int)+96) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #04 pc 0000000000664a94  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanSetup::WaitOnFences(bool)+112) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #05 pc 0000000000664c3c  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanSetup::SubmitCommandBuffer(VkSubmitInfo, bool, bool, bool)+224) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #06 pc 000000000065d788  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanCommandBuffers::EndRecording(VkCommandBuffer_T*, EMyEngineVulkanCommandBufferType, bool)+460) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #07 pc 0000000000664998  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanSetup::EndScene()+596) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #08 pc 000000000065a058  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineGraphicsDeviceVulkan::endScene()+48) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #09 pc 00000000004afdb8  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CFE::Render()+552) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #10 pc 00000000005ed3b0  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (MainLoop()+64) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #11 pc 0000000000637668  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (Java_com_mycompany_myengine_MyEngineJNI_step+36) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #12 pc 00000000000d71e0  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/oat/arm64/base.odex (art_jni_trampoline+144)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #13 pc 00000000001ad8cc  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/oat/arm64/base.odex (com.mycompany.myengine.MyEngineVkSurfaceView$VkThread.guardedRun+2252)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #14 pc 00000000001ae69c  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/oat/arm64/base.odex (com.mycompany.myengine.MyEngineVkSurfaceView$VkThread.run+252)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #15 pc 0000000000133564  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: 05e35aec4fea8e41837577ac976d8eb0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #16 pc 00000000001a8a78  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) (BuildId: 05e35aec4fea8e41837577ac976d8eb0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #17 pc 0000000000553f30  /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithJValues<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, jvalue const*)+460) (BuildId: 05e35aec4fea8e41837577ac976d8eb0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #18 pc 00000000005a32d8  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1308) (BuildId: 05e35aec4fea8e41837577ac976d8eb0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #19 pc 00000000000b0088  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+64) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #20 pc 0000000000050408  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG: deallocated by thread 5546:
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #00 pc 0000000000048edc  /apex/com.android.runtime/lib64/bionic/libc.so (gwp_asan::AllocationMetadata::CallSiteInfo::RecordBacktrace(unsigned long (*)(unsigned long*, unsigned long))+80) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #01 pc 000000000004949c  /apex/com.android.runtime/lib64/bionic/libc.so (gwp_asan::GuardedPoolAllocator::deallocate(void*)+184) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #02 pc 0000000000085d30  /vendor/lib64/hw/vulkan.adreno.so (qglinternal::vkDestroyFramebuffer(VkDevice_T*, VkFramebuffer_T*, VkAllocationCallbacks const*)+96) (BuildId: e52a46a3b09769b61e4e69576af2656e)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #03 pc 000000000065f4e0  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanMemory::FlushQueuedFrees(int)+292) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #04 pc 0000000000664a88  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanSetup::WaitOnFences(bool)+100) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #05 pc 0000000000664c3c  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanSetup::SubmitCommandBuffer(VkSubmitInfo, bool, bool, bool)+224) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #06 pc 000000000065d788  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanCommandBuffers::EndRecording(VkCommandBuffer_T*, EMyEngineVulkanCommandBufferType, bool)+460) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #07 pc 0000000000664998  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineVulkanSetup::EndScene()+596) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #08 pc 000000000065a058  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CMyEngineGraphicsDeviceVulkan::endScene()+48) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #09 pc 00000000004afdb8  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (CFE::Render()+552) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #10 pc 00000000005ed3b0  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (MainLoop()+64) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #11 pc 0000000000637668  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/lib/arm64/libmygame.so (Java_com_mycompany_myengine_MyEngineJNI_step+36) (BuildId: 891b853032c73e98bb00b9399d6860fd2298cbf9)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #12 pc 00000000000d71e0  /data/app/~~B2ZJizd-XsdtobOOB3hhhA==/com.mycompany.mygame-_u1LgC68ewMmVtIVrL_X5g==/oat/arm64/base.odex (art_jni_trampoline+144)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG: allocated by thread 5546:
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #00 pc 0000000000048edc  /apex/com.android.runtime/lib64/bionic/libc.so (gwp_asan::AllocationMetadata::CallSiteInfo::RecordBacktrace(unsigned long (*)(unsigned long*, unsigned long))+80) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #01 pc 00000000000493b8  /apex/com.android.runtime/lib64/bionic/libc.so (gwp_asan::GuardedPoolAllocator::allocate(unsigned long)+368) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #02 pc 000000000003b174  /apex/com.android.runtime/lib64/bionic/libc.so (gwp_asan_calloc(unsigned long, unsigned long)+152) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #03 pc 000000000003ba9c  /apex/com.android.runtime/lib64/bionic/libc.so (calloc+116) (BuildId: 3b0dd94de78a8a796f793e81b7adfbd0)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #04 pc 000000000007fd8c  /vendor/lib64/hw/vulkan.adreno.so (qglinternal::vkCreateFramebuffer(VkDevice_T*, VkFramebufferCreateInfo const*, VkAllocationCallbacks const*, VkFramebuffer_T**)+556) (BuildId: e52a46a3b09769b61e4e69576af2656e)
2020-12-03 15:31:47.176 6161-6161/? A/DEBUG:       #05 pc 0000000000000000  <unknown>

What seems to be happening is that my code is releasing Vulkan resources that are no longer needed as they're associated with an old frame who's fence has finished. I've released a framebuffer, which caused the driver to deallocate some memory. Then I've reset a command pool, which seems to have caused the driver to modify some framebuffer related state, writing to the memory that is no longer allocated. The reported use after free always seems to be "524 bytes into a 1048-byte allocation".

I seem to be able to workaround the issue by changing my code to reset command pools before releasing framebuffer resources, but I believe that this should not be necessary per the Vulkan spec - I believe I should be able to free resources in any order once they're no longer in use.

Device is OnePlus 8T (KB2003), with Android 11, Oxygen OS 11.0.5.6.KB05BA.
GPU: Adreno (TM) 650, deviceID: 100990978.
Vulkan Driver number is 2149535744 (presumably 512.501.0)

Our Vulkan implementation is free of validation errors and is used on multiple titles with large user bases.