Forums - Shared object "libfoo_skel.so" not found

I tried renaming the .so file in /system/lib/rfsa/adsp so that the application would not be able to find it, which resulted in a new error from fopen() before the previous dlopen() error. It seems like adsprpc is actually able to find the .so file, but something else is going wrong when it tries to read it and it is producing a misleading "Shared object not found" error. I'm starting to think this is a Hexagon compiler or linker bug causing dlopen() or another dl*() function to fail, and that the "Shared object not found" error is actually being generated in Qualcomm's adsprpc code for a different error condition than an fopen() problem (i.e. they're printing the wrong error message).

I tried returning immediately from the function I was calling and still got the "not found" error. I then tried #if 0-ing all of the code in a different source file (one where nothing was being called at all) and the error went away. I then put back in only a single line of code:

#include <hexagon_protos.h>

and the error came back again. Unfortunately, there's no source code available for any of the adsprpc stuff as far as I can tell so I don't have any visibility into what might be going wrong, other than possibly that dlopen() is returning an error code of -5.

Anyone have any ideas here? This is currently an urgent problem since we no longer do any development on the aDSP.

Correction: we do use the aDSP. We can no longer do development until this issue is resolved.

Hmm, sounds like you're having an actual problem with it not finding the shared object. Make sure you're setting ADSP_LIBRARY_PATH properly. In our apps we call setenv() in JNI_OnLoad() automatically. I've found that even though the documentation says it should default to /system/lib/rfsa/adsp, it doesn't seem to work when called from an Android (Java) application.

Yeah I've found that same issue. I think I have all that working now - it loads and works when I remove certain code that should be innocuous, but not when I put the code back in, even though the code is never executed.

I'm fairly sure there's something wrong with the Hexagon compiler or linker, and/or with the adsprpc dlopen() implementation. Unfortunately that code is Qualcomm-proprietary. It would be helpful if someone from Qualcomm would notice and respond to this thread. That doesn't seem to happen very frequently on this forum unfortunately.

Hi Himateja,

I was not able to resolve the issue yet - the library was named correctly and is definitely being found (note the mention above of the fopen() error) but I'm experiencing issues that I think are due to the tools. I initially suspected alignment problems or something similar (I've seen similar issues due to compiler bugs in other platforms in the past) so I consulted the ABI and used objdump to do some more investigating. I was able to make some progress as follows:

1. I narrowed the issue down to the declaration of a static uint32 array (in global space, not on the stack) containing FFT twiddles from the Hexagon Tools optimzed FFT example that had 3072 entries (12288 bytes). I found the point where the issue stopped by reducing the number of array entries - the problem went away at 2062 entries (8248 bytes). I initially thought it was an alignment issue or a size issue going above 0x2000, but 8248 is 0x2038 bytes and I confirmed that the array was correctly aligned using the aligned attribute and by checking its address with objdump. Note that at all times I was not accessing this array. I was simply calling a single aDSP function through adsprpc that returned immediately.

2. I had a theory that there were still memory alignment or overlap issues and I noticed the debug sections were right near the end of the .data section in objdump. I disabled debug sybols (turned off -g3 for C files, which is on by default) and the issue stopped happening. I don't know what's wrong with the compiler that debug functions is causing this issue, or why adsprpc is giving the error "Shared object '*_skel.so' not found" instead of something more accurate/helpful.

3. Now a new issue showed up. I was attempting to pass a struct into an adsprpc function. The struct and function taking the struct as rout were declared properly in the IDL file and were compiling properly, but when I tried to call the function in adsprpc, passing in a struct, I got an error "Unresolved PLT function 'free'". I looked into the *_skel.so file and noticed that the following symbols were unresolved: malloc, free, memset, memcpy, all of which should be in libc. I removed the struct function from the IDL and the use of malloc and free went away. memset and memcpy were still unresolved in objdump, but they opened just fine when dlopen() loaded the library (note again that the function using them wasn't called so I don't know if it would hit other errors later). I noticed that free() is called by the pack/unpack functions generated in the *_skel.c file to marshall the struct.

As you can see, I'm still having pretty tough issues that I think are tools-related. I would definitely appreciate help. Are there any debug prints that can be enabled in adsprpc? Is there any other information I can provide?

In the meantime I have one more possibility I can try to resolve this issue. I'm going to try to work around the problem by creating a persistent thread on the aDSP. I've never done that before so I don't know if I'll have other issues, or if the previous issues will arise again, but we'll see.

Thanks,

Adam

Sorry for the triple reply. The (new?) reply approval process threw me for a loop. Not entirely sure where that came from.

I just wanted to add one more question - where are new SDK releases posted? I wasn't aware that there had been a new release since 1.1.0, and I don't have any additional downloads available in my Me2U inbox. I'm not sure whether or not the the DragonBoard BSP supports the new SDK, but it would definitely be nice to have any updates or bug fixes it might include, especially any improvements to the tools.

David,

Have you tried disabling debug symbols? That resolved my issue and highlighted the new issue (undefined symbol 'free()') that arises when a function takes a struct as a parameter in the QAIC IDL. I'm still trying to sort out that issue or find a workaround, so I haven't actually tested the original code on the device yet, but the dlopen errors are different now so I'm hoping that means they're resolved.

Adam

Ah, very helpful!

I switched to a command line build from an Eclipse build this morning and noticed that the unresolved symbol errors went away when sending structs over. Looking at the verbose output, it looks like that's exactly what it's doing. I'll have to try adding these wrap options to Eclipse once I get things functioning a bit.

Do you know if that has any bearing on the original issue though? I'm not seeing it currently, but I'm no longer convinced that disabling debug symbols was what solved it.

On another note, looking into the stub/skel code it looks like the code is performing copies of all of the data in my arrays I'm passing back and forth. I thought adsprpc was supposed to be zero-copy if you allocate buffers through ION? What is the correct way to pass buffers back and forth, and how can I get zero-copy to work? I have a realtime function that performs a number of operations on a stream of input samples, returning some number of results. I'd like the input and output to be zero copy for efficiency reasons. I'd also like to allocate all of the intermediate buffers the function needs with ION on the host side and pass pointers to them into the aDSP.

One way I've considered is to do this with a persistent thread that allocates all of its own internal buffers, but I'm not entirely sure what I need to do to link in QuRT support. Looking at the downscale example it seems like there are at least a few static libraries that need to be linked into the host application, but I would have thought you would link stuff to the skel library? It's also not clear to me which of those libraries are actually needed/used and what they contain. I was able to build a skel with qurt calls in it, but when I run it from my Android app (with none of the static libraries linked in) I get undefined references to HAP_pls_add_lookup(). See also https://developer.qualcomm.com/forum/qdn-forums/add-advanced-features/computer-vision-fastcv/27532.

Thanks,

Adam

Update: I am still seeing the original issue (shared object not found), which seems to be occuring when I include a large static global array (the array of FFT twiddles included in the Hexagon Tools examples). This wasn't occurring earlier today, though the array was compiled in, because I had removed a large chunk of code in an effort to get things functioning incrementally. As soon as I added back enough code, the error came back. Turning off debug symbols no longer seems to fix the issue. Note: This issue seems to exist when building with either the command line or Eclipse.

I tried removing the static initializer so that the array would be put into .bss instead of .data, thinking that there might be an issue with it showing up at the end of .data, but that didn't make much difference one way or another. I did notice that the issue also goes away if I comment out the call to fft16x16 (the assembly-optimized complex FFT kernel from Hexagon Tools). If I leave the call in but add an early return so the function never gets called, however, I still get the error.

As always, assistance is greatly appreciated. Without the FFT call I can't move forward at all so I'm at a standstill at the moment.

Update: I was able to work around this limitation by allocating everything dynamically and storing the buffers in static pointers. This wasn't ideal, but seems to be working. I find when the global memory map (.data or .bss) gets near 128k (0x20000) I start to see the error.