I'm working on getting camera images for a downward facing VIO camera and two forward facing VOA cameras. We have a HAL3 application which is exposing a 4k camera on a gimbal for the user to view video and record video and take snapshots.
We have a separate application which we worked on with a partner which does VIO and feeds information into our PX4 flight system. We found that if we try to instantiate multiple HAL3 applications the second one always segfaults. This is true even if we just run hal3_test twice. So I began researching new packages in our yocto distribution. I found the camera-bridge package in the meta-qti-robotics layer. This produced a camera1-test app which relies on a cameraserver application to be running. I found that when a HAL3 application is running the cameraserver application also crashes and when I run either of them the log shows that they're opening the same resources. I've been looking in create point and when I search on sda845 and robotics there are over 2200 documents. I've done some filtering and searching through there and I think I must need to create a native chi application so that I can bypass the HAL3 and the ISP's? Additionally there are only 2 ISP and I've got 4 cameras, only one of which I actually want going through the ISP and HAL3. I know others have implemented many more cameras than I'm even attempting.
Can anyone point me at the proper software, packages, or documentation so that I can get images from my VIO and VOA cameras while bypassing the ISP and HAL3 which is in use by our user facing 4k camera. Or perhaps my entire approach is wrong? Any comments or help is appreciated.
I am unsure what you mean by VIO or VOA cameras and "a partner who does VIO". Is this a question about cameras that feed into a MV SDK feature like mvVISLAM?
Thank you for your prompt reply and I apologize for the confusion! We have a downard facing camera for Visual Inertial Odometry (VIO) and two forward facing cameras for Vehicle Obstical Avoidance (VOA) (Not a robotics expert but I think this is not full SLAM, but pieces of it).
Our partner has the algorithms to analyze the frames and provide the feedback to PX4. Their software is currently using HAL3 which has led to this conflict that I can not run our 4k camera through gstreamer and the VIO software at the same time.
My goal is to understand how I can get frames from these three cameras for our VIO software (and future VOA), but bypass the ISP's and HAL3 layer. Unless my approach is completely wrong, which I'm open to feedback. Does the Machine Vision SDK software get frames / images from cameras using the HAL3 or does it grab the image data using CHI SDK direclty (I think this is what I need to do)?
There are other camera and image packages in my yocto distribution but the one that looked promising (camera-bridge) has the same conflict and segfault when I run it at the same time as our HAL3 application. I'm wondering from an architecure point of view if we've missed something? Should we be able to run multiple instances of applications that access cameras through the HAL3 layer, one that provides video for users and one or more that are for SLAM, VIO, or VOA?
No, the MV SDK does not get frames from cameras but does have some sample applications (e.g., "app/viSLAM") that might serve as some example. It depends on which board you are using as to how many cameras are supported but only due to OS limitations really. For CV/MV applications, one can always bypass the frameworks entirely and just talk to the sensors directly which is the lowest latency path anyway. One can then use mvCPA in the loop to set gain/exposure values.
Thanks. Bypassing the frameworks is and getting sensor data directly is exactly what I think I need to do (I'm still learning). I probably am using the wrong nomenclature with the work "frame". I'll investigate the MV SDK sample applications.