Skill Level | Area of Focus | Operating System | Platform/Hardware |
---|---|---|---|
Intermediate | Computer Vision, Embedded, Robotics, Sensors, Smart Cities | Ubuntu / Ubuntu Core | Qualcomm Robotics RBx Dev Kit |
Objective
The main objective of this project is to start your artificial intelligence (AI) application development for robotics using the Qualcomm Robotics RB5 development kit. It will walk you through the following steps:
- Compile the TensorFlow lite libraries
- Test USB camera
- Run an objection detection application on the CPU of the Qualcomm Robotics RB5 development kit.
Materials Required / Parts List / Tools
- An Ubuntu 18.04 PC
- Qualcomm Robotics RB5 Development kit
- A USB camera
- A display monitor
Source Code / Source Examples / Application Executable
Build / Assembly Instructions
Hardware set up.
- Connect the Qualcomm Robotics RB5 development kit to the monitor through HDMI cable.
- Plugin a keyboard and a mouse to the development board.
- Connect the USB camera module to the development board.
2. Wi-Fi setup on Qualcomm Robotics RB5 Development kit
Use the below command to setup the Wi-Fi connection.
root@qrb5165-rb5:/# vi /data/misc/wifi/wpa_supplicant.conf
Please modify the details listed below to set Wi-Fi host name and password,ten save it and reboot the device.
network={ssid="YOUR_SSID_NAME"key_mgmt=WPA-PSKpairwise=TKIP CCMPgroup=TKIP CCMPpsk="PASSWD"}
After the reboot is finished, run the command “ifconfig” to check if the Wi-Fi is connected.
3. Run Weston desktop on Qualcomm Robotics RB5 Development kit
If the HDMI cable connects properly, after the device boot up, you will see the device boot up logs and then see a shell is ready for command input. Run the following command to launch Weston desktop.
root@qrb5165-rb5:/# Weston --connector=29
If it fails to run, you can try the following commands with adb on an Ubuntu PC. Please make sure the Qualcomm Robotics RB5 development kit is connected to the Ubuntu PC with a USB cable. Please ignore this step if the above command runs successfully.
adb shellmkdir -p /usr/bin/weston_socketchmod 700 /usr/bin/weston_socketexport XDG_RUNTIME_DIR=/usr/bin/weston_socketexport LD_LIBRARY_PATH=/usr/lib:/usr/lib/aarch64-linux-gnu/weston --tty=1 --connector=29 --backend=drm-backend.so
4. Build the TF lite library on the Ubuntu host machine.
Here are the commands. For more information see the link:
https://www.tensorflow.org/lite/guide/build_arm64git clone https://github.com/tensorflow/tensorflow.gitcd tensorflowgit branch r2.1 remotes/origin/r2.1git checkout r2.1./tensorflow/lite/tools/make/build_generic_aarch64_lib.sh
Please make sure the libtensorflow-lite.a is generated after the above steps are done.
5. Build OpenCV for Wayland
The Qualcomm Robotics RB5 development board supports Wayland/Weston as the display server, this OpenCV is a special version for Wayland compiled on the development board.
For More info about OpenCV refer:
https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html.Here are the steps to compile OpenCV on the target Qualcomm Robotics RB5 Development kit
root@qrb5165-rb5:/# sudo apt-get install build-essential curl unziproot@qrb5165-rb5:/# sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-devroot@qrb5165-rb5:/# sudo apt-get install libjpeg-dev libpng-devroot@qrb5165-rb5:/# sudo apt-get install python-numpy libxkbcommon-dev libwayland-client0 libwayland-devroot@qrb5165-rb5:/# mkdir /home/srcroot@qrb5165-rb5:/# cd /home/srcroot@qrb5165-rb5:/# git clone https://github.com/pfpacket/opencv-wayland.gitroot@qrb5165-rb5:/# cd opencv-wayland/root@qrb5165-rb5:/# mkdir buildroot@qrb5165-rb5:/# cd buildroot@qrb5165-rb5:/# cmake -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/usr/local -DWITH_IPP=OFF -DWITH_WAYLAND=ON -DWITH_GTK=OFF ..root@qrb5165-rb5:/# make -j7
6. Build the object detection application on target
Qualcomm Robotics RB5 Development kit.
Before you build the application, install TFlite dependencies first.
Here are commands to install TFlite dependencies.
root@qrb5165-rb5:/# cd /home/srcroot@qrb5165-rb5:/# git clone https://github.com/tensorflow/tensorflow.gitroot@qrb5165-rb5:/# cd tensorflowroot@qrb5165-rb5:/# git branch r2.1 remotes/origin/r2.1root@qrb5165-rb5:/# git checkout r2.1root@qrb5165-rb5:/#./tensorflow/lite/tools/make/download_dependencies.sh
The following steps are to build the application.
root@qrb5165-rb5:/# cd /home/srcroot@qrb5165-rb5:/# git clone https://github.com/mattn/webcam-detect-tflite.gitroot@qrb5165-rb5:/# cd webcam-detect-tfliteroot@qrb5165-rb5:/# mkdir libs
Place the libtensorflow-lite.a file generated on the step #4 to the libs folder using the command adb push libtensorflow-lite.a
Copy the following content and save it as CMakeList.txt to /home/src/webcam-detect-tflite
cmake_minimum_required(VERSION 3.0)project(webcam-detector)set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17 -O2 -g")set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ldl -lrt")message(STATUS "optional:-std=c++17")#set(OpenCV_DIR "/home/src/opencv-wayland/build/")include_directories(/usr/include/freetype2/)include_directories(/home/src/tensorflow/)include_directories(/home/src/tensorflow/tensorflow/lite/tools/make/downloads/flatbuffers/include/)find_package(OpenCV REQUIRED)message(STATUS "OpenCV library status:")message(STATUS " version: ${OpenCV_VERSION}")message(STATUS " libraries: ${OpenCV_LIBS}")message(STATUS " include path: ${OpenCV_INCLUDE_DIRS}")include_directories( ${OpenCV_INCLUDE_DIRS} )ADD_LIBRARY(TFlite_LIB STATIC IMPORTED)SET_TARGET_PROPERTIES(TFlite_LIB PROPERTIESIMPORTED_LOCATION /home/rb5/webcam-detect-tflite/libs/libtensorflow-lite.a)set(SOURCE_FILESmain.cxx)add_executable(${PROJECT_NAME} ${SOURCE_FILES})target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} )target_link_libraries(${PROJECT_NAME}freetypepthreaddl)target_link_libraries(${PROJECT_NAME} TFlite_LIB)
After saving the above content to CMakeList.txt, execute the following commands:
root@qrb5165-rb5:/# mkdir buildroot@qrb5165-rb5:/# cd buildroot@qrb5165-rb5:/# cmake ..root@qrb5165-rb5:/# make
If the above steps run successfully; the application webcam-detector will be generated under the folder build.
If you want to know more details about Tensorflow lite, please visit:
Usage Instructions
Before running the application, please check the USB camera node.
By default, the device node in kernel should be /dev/video2. You can reconnect the USB camera module to check if this node appears and disappears accordingly.
Run the following commands on the Qualcomm Robotics RB5 Development kit to run the application.
root@qrb5165-rb5:/# cd home/src/webcam-detect-tflite/buildroot@qrb5165-rb5:/# ./webcam-detector
Then you will see the classification result as shown below.

Contributors
Name | Title/Company |
---|---|
Weizhang Luo | Senior Staff Engineer, Qualcomm Communication Technologies (Shenzhen) Co., Ltd. |
Qualcomm Robotics RB5 and Qualcomm QRB6165 are products of Qualcomm Technologies, Inc. and/or its subsidiaries