Hello, I have been told to use Gstreamer to feed camera feeds into an snpe mode for inference. How would I do this with ROS2?
Thanks.
Gstreamer and SNPE with ROS2
Posted: Mon, 2021-08-23 07:34
Forums - Gstreamer and SNPE with ROS2
Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.
qtimlesnpe - Qualcomm Developer Network
Qualcomm Neural Processing SDK for AI - Qualcomm Developer Network
Here are some links that describe how gstreamer and SNPE work together. And a link to the related gstreamer snpe plugin.
Application Semantics - Qualcomm Developer Network
This gives you architecture details on the relationship between ROS2 - SNPE and GST
For some reason I can't post a direct link but if you look at the bottom of the architecture page it has a link to gstreamer sample code. Its not specific to SNPE or ROS2 but it helps get you started.
hello ciustomer,
Please describe the use case.Do you use the ros2 system on the RB5 device and then use the GStreamer?
Hello Mita,
Thanks for getting back to me. The use case is as follows:
We need to capture 2 onboard MIPI camera feeds (we were doing it with OpenCV on the Host PC but it didn't work with the onboard cameras on the RB5 and we were told to use GStreamer instead). We then need to be able to process and combine these camera feeds (we were doing it using cv::Mat and std::vector). We finally have to feed these processed tensors into the SNPE for inference, potentially converting whichever processed tensor we end with into a USERBUFFER tensor to feed into the SNPE and then extracting another tensor from inferene to work with afterwards.
It would be great if you could detail howyou would achieve each of those steps. We don't need ROS2 per se, we only use ROS2 for middleware communications, but if it can be done without ROS2 we are happy to explore that option.
Kind regards,
Dear customer,
Please check the examples from GStreamer: https://gstreamer.freedesktop.org/documentation/tutorials/basic/index.html?gi-language=c
QC made the SNPE to GStreamer component, too. So you can link it to the GStreamer.