Forums - Build OpenCV and SNPE on rb5

20 posts / 0 new
Last post
Build OpenCV and SNPE on rb5
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Wed, 2021-08-25 03:05

Hello,

Which are the correct OpenCV libraries I should link with the target SNPE executable on the RB5 and where can I find these libraries? (I have downloaded the OpenCV source code but can't see any aarch64_linux libraries to link to)

Thanks

 

  • Up0
  • Down0
jlowman
Join Date: 25 Aug 21
Posts: 16
Posted: Thu, 2021-08-26 15:41

Object detection with TensorFlow Lite - Qualcomm Developer Network

If you look at step 4, it has instructions for building OpenCV for Weyland on the RB5.   It done on target (so buld on the board).

 

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Mon, 2021-08-30 06:29

Hello and thank you for your answer.

I tried following the example in your link but I get the follwing error:



 /home/src/webcam-detect-tflite  ./webcam-detector
Loading model: mobilenet_quant_v1_224.tflite

(process:8785): GStreamer-CRITICAL **: 13:24:22.731: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed
OpenCV | GStreamer warning: GStreamer: unable to query pipeline state (/home/src/opencv-wayland/modules/videoio/src/cap_gstreamer.cpp:421)
(DEBUG) try_init_v4l2 VIDIOC_QUERYCAP "/dev/video0": Inappropriate ioctl for device
Unable to stop the stream: Inappropriate ioctl for device
Failed to open VideoCapture.
 

How can I solve this errror?

Thanks in advance.

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Mon, 2021-08-30 07:45

If I change the api in the OpenCV code in main.cpp from cv::CAP_ANY to cv::CAP_V4L2, I get the follwing error:

(DEBUG) try_init_v4l2 VIDIOC_QUERYCAP "/dev/video0": Inappropriate ioctl for device
Unable to stop the stream: Inappropriate ioctl for device

Please help me get rid of this error, or plrease update your example in https://developer.qualcomm.com/project/object-detection-tensorflow-lite#.... to work properly.

Thanks

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Tue, 2021-08-31 00:33

If I try to build OpenCV Wayland with

-D WITH_LIBV4L=ON

I get:

Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device
VIDEOIO ERROR: V4L: device /dev/video0: Unable to query number of channels

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Tue, 2021-08-31 01:39

If I try to list the video devices in the RB5, I get the following:


sh-4.4# ls -ltrh /dev/video*
crw-rw---- 1 root video 81,  1 Feb 23  2021 /dev/video1
crw-rw---- 1 root video 81, 33 Feb 23  2021 /dev/video33
crw-rw---- 1 root video 81, 32 Feb 23  2021 /dev/video32
crw-rw---- 1 root video 81,  0 Feb 23  2021 /dev/video0
sh-4.4# sudo apt-get install v4l-utils
Reading package lists... Done
Building dependency tree       
Reading state information... Done
v4l-utils is already the newest version (1.14.2-1).
0 upgraded, 0 newly installed, 0 to remove and 106 not upgraded.
sh-4.4# v4l2-ctl --list-devices
msm_vidc_vdec ():
    /dev/video32
    /dev/video33
    /dev/radio0
    /dev/v4l-subdev2
    /dev/v4l-subdev3
    /dev/v4l-subdev4
    /dev/v4l-subdev5
    /dev/v4l-subdev6
    /dev/v4l-subdev7

VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
/dev/video0: not a v4l2 node
sh-4.4# sudo v4l2-ctl --device=/dev/video0 --all
VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
/dev/video0: not a v4l2 node
sh-4.4# sudo v4l2-ctl --device=/dev/video1 --all
VIDIOC_QUERYCAP: failed: Inappropriate ioctl for device
/dev/video1: not a v4l2 node
 

  • Up0
  • Down0
jlowman
Join Date: 25 Aug 21
Posts: 16
Posted: Tue, 2021-08-31 13:18

I think you'd have to post these questions in the software AI forum, they know a lot more about those tools and how to use them than I do.

  • Up0
  • Down0
pe.ostad
Join Date: 21 May 21
Posts: 14
Posted: Fri, 2021-09-03 00:45

Hi,

Are you using a USB Camera? Because this example works with a USB camera, not the onboard cameras.

You should use Gstreamer to get access to the cameras using qtiqmmfsrc plugin.

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Fri, 2021-09-03 06:03

Hello and thank you vrey much for your answer.

I was using the onboard MIPI cameras. So you are saying that the only way to get a camera feed from the MIPI cameras is through the GStreamer plugin? I am sure I have used V4L2 to access MIPI cameras on other platforms.

Why is GStreamer the preferred option?

Can the Gstreamer be invoked from within a C++ main.cpp file or does it have to be invoked from the terminal and have it running in the background and then somehow redirect he camera feed to the main.cpp application?

Can you procide an example of how you would do it?

Thanks in advance.

  • Up0
  • Down0
jlowman
Join Date: 25 Aug 21
Posts: 16
Posted: Fri, 2021-09-03 10:51

GitHub - quic/sample-apps-for-Qualcomm-Robotics-RB5-platform

In the RB5 github there are gstreamer code samples

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Mon, 2021-09-06 03:28

Hello, I had come across the GStreamer applications, but I would like to know if it is the only option to work with the onboard MIPI cameras, meaning V4L2 would not be an option.

I kindly ask you to reply to my quesions above too.

Kind regards.

  • Up0
  • Down0
SahilBandar
Join Date: 23 May 18
Posts: 37
Posted: Tue, 2021-09-07 20:38

Hi 22imonreal,

The camera which is connected on CAM port of RB5 won't support directly with VideoCapture with camera id.
Reason for that is ioctl call for Camera drivers of RB5's Camera  is not matching with the V4L2's ioctl call.

For making it work out, you can create the gstreamer pipeline for the camera using qtiqmmfsrc element for input source.

Please follow the steps in order to resolve your problem,

You can stream the video on TCP using Gstreamer & process by OpenCV's VideoCapture
 
Open shell of RB5 & run following command for creating the TCP Streaming Pipeline,
gst-launch-1.0 -e qtiqmmfsrc name=qmmf ! video/x-h264,format=NV12, width=1280, height=720,framerate=30/1 ! h264parse config-interval=1 ! mpegtsmux name=muxer ! queue ! tcpserversink port=8900 host=<RB5_IP_ADDRESS>

NOTE: Please keep running the above pipeline in seperate terminal.

Now, you can use VideoCapture, and pass streaming URL to get the frames using OpenCV, Please refer below Example,

import cv2

cap = cv2.VideoCapture("tcp://<RB5_IP_ADDRESS>:8900") #rb5 ip & port (same from command)
while(cap.isOpened()):
    ret, frame = cap.read()
    
    #
    # Write your code here
    # 

    break
cap.release()
 

Hope this will resolve your issue,

Thanks & Regards,
Sahil Bandar

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Wed, 2021-09-08 08:26

Thank you very much for your answer, it definitely helps!

I tried saving the frame to a file as follows, but I got the following error:



[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] non-existing PPS 0 referenced
[h264 @ 0x5603af5c24c0] decode_slice_header error
[h264 @ 0x5603af5c24c0] no frame!
 

Script:

import cv2

cap = cv2.VideoCapture("tcp://192.168.1.104:8900") #rb5 ip & port (same from command)
while(cap.isOpened()):
ret, frame = cap.read()
 
#
# Write your code here
#
cv2.imwrite("./last_frame.jpg", frame)

break
cap.release()

Any suggestions as to why I wouldn't be able to save the frame?

Thank you in advance.

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Wed, 2021-09-08 08:28

Actually, I have checked that the frame is correctly saved as a .jpg image, so I will try to embed that code inside a main loop and see if it no longer fails with a missing frame.

Any other suggestions are welcome.

  • Up0
  • Down0
SahilBandar
Join Date: 23 May 18
Posts: 37
Posted: Wed, 2021-09-08 09:02

Hi 22imonreal,

The msg logs which you are getting is from h264 decoder, in start of the application, because of empty buffer it will give the same warning, and it won't affect your execution of application.
For few set of frame in start there are chances that you will get this warning, but it will work fine after 2-3 seconds of starting the application.

If you are getting empty image as a written in storage, Please remove the break from while loop of your source, It is causing the break loop and storing just first frame in loop which can be blank also. 

Thanks & Regards,
Sahil Bandar

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Fri, 2021-09-10 08:13

Hello Sahil,

Thank you for your help.

For some reason I cannot get it to work in C++. Would you be able to provide an equivalent piece of code to the Python code provided above but in C++ this time?

Thanks

  • Up0
  • Down0
22imonreal
Join Date: 10 Feb 21
Posts: 80
Posted: Sun, 2021-09-12 02:01

With this application:

#include <iostream>
#include <stdio.h>

 

#include <opencv2/core.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>

 

int main()
{

 

uint count = 0;
std::string camera_ip = "192.168.1.104";
cv::VideoCapture cap("tcp://192.168.1.104:8900");
cv::Mat frame;

 

if (!cap.isOpened()) {
std::cerr << "ERROR! Unable to open camera! " << camera_ip << "\n";
//return -1;
}
else
{
while (cap.isOpened())
{
cap.read(frame);
if (frame.empty())
{
std::cerr << "ERROR! Blank frame grabbed! \n";
}
else
{
cv::imwrite("../data/input/raw/mipi/frame_"+ std::to_string(count) +".jpg", frame);
std::cout << "SUCCESS! Writing image " << count << "\n";
//std::system("python /home/inigo/dev/osd/rb5/osd_snpe_inference/tools/to_raw.py");
if (count > 25)
{break;}
else
{count += 1;}
}
}
}
cap.release();
return 0;
}


I'm getting the following error:

OpenCV | GStreamer warning: incompatible gstreamer version (/opencv-wayland/modules/videoio/src/cap_gstreamer.cpp:130)
OpenCV | GStreamer warning: your gstreamer installation is missing a required plugin
 (/opencv-wayland/modules/videoio/src/cap_gstreamer.cpp:1827)
OpenCV | GStreamer warning: GStreamer: unable to start pipeline
 (/opencv-wayland/modules/videoio/src/cap_gstreamer.cpp:873)

(grab_isp_cam:11150): GStreamer-CRITICAL **: 08:58:05.068: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed
OpenCV | GStreamer warning: GStreamer: unable to query pipeline state (/opencv-wayland/modules/videoio/src/cap_gstreamer.cpp:421)
[tcp @ 0x558c8f80b0] Connection to tcp://192.168.1.104:8900 failed: Connection refused
warning: Error opening file (/opencv-wayland/modules/videoio/src/cap_ffmpeg_impl.hpp:834)
warning: tcp://192.168.1.104:8900 (/opencv-wayland/modules/videoio/src/cap_ffmpeg_impl.hpp:835)
VIDEOIO ERROR: V4L: device tcp://192.168.1.104:8900: Unable to query number of channels
ERROR! Unable to open camera! 192.168.1.104

Why does the error suggest I am missing a required plugin? What required plugin?

Does the above have anything to do with building OpenCV Wayland with -DFFMPEG=ON?

Thanks for your help.

  • Up0
  • Down0
SahilBandar
Join Date: 23 May 18
Posts: 37
Posted: Thu, 2021-09-16 05:25

Hi 22imonreal,

I've tested the both CPP & Python Script and in both case this is working fine for me on RB5.

As You can see in the error which you posted, It is saying connection refused to 192.168.1.104:8900, that means either you have not started the gstreamer pipeline in seperate terminal or not connected to network or you have entered the wrong IP of RB5.

Please check the aspect mentioned above to resolve the problem.

Hope this will solve your problem.

Thanks,
Sahil Bandar

  • Up0
  • Down0
codestacy
Join Date: 27 Nov 21
Posts: 1
Posted: Sat, 2021-11-27 01:54
Main difference OpenCV is computer vision library and TensorFlow is Machine Learning Tool. definition, differences, application, languages they support
 
  • Up0
  • Down0
karishma.inamdar
Join Date: 12 Oct 22
Posts: 3
Posted: Fri, 2023-01-06 17:20

I am having same set of problems and cross checked evrything. How can I resolve this issue? My Opencv-wayland doesn't have buildin Gstreamer. How to install one?

  • Up0
  • Down0
or Register

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.