Hello,
My company is about to release a game on Android, both on Google Play and Amazon Shop. It should have been released before the last weekend but it has been delayed for the reason I am about to describe.
It is a remake of a PC point-&-click game released circa 2001, so it is graphically pretty intensive (a fullscreen background is loaded each time you enter a scene, as well as videos and other objects, with 3D characters on top) and we don't have much freedom in the way we can reduce the number of assets if we want to stay faithful to the original. As a side note, the game is 1.3GB big, which is a problem in itself for the Amazon version on some devices.
Our problem is, pretty often (every 10-15 minutes of gameplay), the game crashes on Kindle Fire HDX with this message in the logs:
Adreno-GSL <sharedmem_gpumem_alloc_id:1393>: sharedmem_gpumem_alloc: mmap failed errno 12 Out of memory
Adreno-GSL <ioctl_kgsl_sharedmem_alloc:1494>: ioctl_kgsl_sharedmem_alloc: FATAL ERROR : (null)
The Kindle Fire HDX is a pretty meaty device (SnapDragon 800 and Adreno 330), and we don't have this problem on any other device we tested (Galaxy Note I and II, Galaxy S III and IV, Galaxy Tab I and II, Kindle Fire HD, even Archos ARNOVA 7 G2), all of which hold either Mali or PowerVR GPUs. Our engine has been checked for texture leaks, and all the games that use it can run on an iPad 1 without problem. That is what makes me think it is a driver problem.
Is there some kind of optimization in the ways the textures are handled? (If we move around the same 2 or 3 locations for half an hour, there is no problem, while if we play a real walkthrough, we can expect a crash every 15 minutes, and in both cases, in our engine, all textures are unloaded each time you leave a scene) Is there a way to make sure textures are REALLY deleted? I have not been able yet to try the Adreno SDK, since we are in a hurry and this problem popped up at the last moment.
Thanks in advance for your time,
Guntha
Good morning,
I just had a chance to use the Adreno Profiler, and things are getting even more confusing.
I tried to play for a little while, syncing textures each time I entered a scene. And the profiler clearly showed me that only the textures from the scene in which I was were allocated, so there is no texture leak (according to the profiler).
I could provide the textures provided by the profiler, as well as the test scenario that lead to this export, if necessary.
Guntha
Keep in mind that the Adreno Profiler is doing a desktop emulation of the driver (a somewhat different rendering path). Could you provide us with an apk to verify the problem? It could be a driver problem that we could take a closer look at. Also deleting the textures via OpenGL calls should do the trick. Assuming the textures are reasonably sized (2kx2k or less) and compressed?
When the app crashes, the logcat could provide some clues as to where the problem lies.
Hello mhfeldma,
The problem is that the apk is pretty big (1.3GB). We also have a version with a separated obb file for Google Play. Is there a convenient way for us to provide you with it?
Also, I recently discovered that the app leaks on every device, except the leaks on the HDX seem much bigger. The difference is such that on most device we can make a complete walkthrough, while on the HDX it crashes every 10-15 minutes. We did release the app on Google Play, and we have a customer feedback that says the app crashes every 15 minutes on a Galaxy S3 (We have a S3 here, and we didn't have this problem. Howevere there are versions of the S3 that hold an Adreno instead of a Mali).
As a side note, there is no texture above 1024x1024, they are uncompressed, they are created mostly from jpeg (for storage purposes) and png files, and they are always deleted via glDeleteTextures. I tried to add, right before the delete, a glTexImage2D with a zero-sized image and a null buffer as parameters, to see if it could make any difference (it did not). I also checked if the glDeleteTextures could be called outside of the thread that created the current egl context. Since our engine is mostly single-threaded, this is not likely to happen.
According to the call stack from our logs, the app crashes right after the FATAL ERROR from the driver: during gameplay, we call glTexImage2D, the driver crashes (or so it seems) during a call to ioctl_kgsl_sharedmem_alloc, calls the exit function from the libc, then the program makes a segfault when we try to delete an element from an array.
Thanks for your answer,
Guntha
After further testings, my leaks are actually roughly the same on other devices. The difference is that it seems the native heap has a limited size on Adreno 330 (around 130MB) while there is no limit on other devices, the latter being the normal behaviour on Android according to my searches. Or maybe there is a fixed size dedicated for video memory on those devices (if the leak is really about textures). Either way, it seems the problem is entirely on my side. I still haven't found a working solution yet.
Well, several things have come up since my last post:
1)What I thought were massive memory leaks was actually the way assets inside the "assets" folder of the apk is managed: for every new piece of data read, Android tries to keep it in RAM for further reads until the file is closed, which explains the leak at each and every new scene, since we use a single packaged file inside the apk (storing each individual file in the apk kills the performance for an app that doesn't preload everything at the beginning). I didn't have an occasion to profile Android memory management before, so I got confused with the Java Heap, the Native Heap and the PSS. This one appeared only in the PSS (when using the meminfo command). The Java Heap stayed constant, and the Native Heap progressed much slower. I "fixed" this by closing the package between each scene.
2) What I thought was a small leak in the Native Heap was actually a cache system for animations of 3D models (Up to 4 MB of animations can be cached) (I'm not the only one working on this project, and this system has been added for the PS3 version of our game: on this platform, disk accesses looked really expensive). Maybe there's still something leaking around here, but it can't be found on other platforms (PC and consoles), and the Native Heap Allocated Size stays pretty constant.
Is it common that a game crashes on Adreno when using around 56 MB of RAM (I'm talking only about Native Heap)? It always crashes on a call to glTexImage2D().
When debugging, when I print the Native Heap informations through getNativeHeapSize(), getNativeHeapFreeSize(), getNativeHeapAllocatedSize(), I am surprised that getNativeHeapSize() is never the sum of getNativeHeapFreeSize() and getNativeHeapAllocatedSize(). I couldn't find on the web what really represents getNativeHeapFreeSize(), what is its behaviour supposed to be? I tried allocating 100MB then deleting it to increase the native heap size, after doing that the total heap size really has increased, but the free heap size always stays around 2MB.
Currently, I am suspecting that the size of this "free size" prevents me from using textures of 1024*1024 or above safely. I thought of this because of a last minute thing: it used to crash on some memory-intensive scenes, and after some modifications on some data, it crashed at another point, in an almost empty scene.
I'm thinking again of providing an apk to Qualcomm support.