Skill Level | Area of Focus | Operating System | Platform/Hardware |
---|---|---|---|
Intermediate | Computer Vision, Embedded, Robotics, Sensors, Smart Cities | Ubuntu / Ubuntu Core | Qualcomm Robotics RBx Dev Kit |
Objective
The main objective of this project is to start your artificial intelligence (AI) application development for robotics using the Qualcomm Robotics RB5 development kit. It will walk you through the following steps:
- Compile the TensorFlow lite libraries
- Test USB camera
- Run an objection detection application on the CPU of the Qualcomm Robotics RB5 development kit.
Materials Required / Parts List / Tools
- An Ubuntu 18.04 PC
- Qualcomm Robotics RB5 Development kit
- A USB camera
- A display monitor
Source Code / Source Examples / Application Executable
Build / Assembly Instructions
Hardware set up.
- Connect the Qualcomm Robotics RB5 development kit to the monitor through HDMI cable.
- Plugin a keyboard and a mouse to the development board.
- Connect the USB camera module to the development board.
2. Wi-Fi setup on Qualcomm Robotics RB5 Development kit
Use the below command to setup the Wi-Fi connection.
[email protected]:/# vi /data/misc/wifi/wpa_supplicant.conf
Please modify the details listed below to set Wi-Fi host name and password, then save it and reboot the device.
network={ssid="YOUR_SSID_NAME"key_mgmt=WPA-PSKpairwise=TKIP CCMPgroup=TKIP CCMPpsk="PASSWD"}
After the reboot is finished, run the command “ifconfig” to check if the Wi-Fi is connected.
3. Run Weston desktop on Qualcomm Robotics RB5 Development kit
If the HDMI cable connects properly, after the device boot up, you will see the device boot up logs and then see a shell is ready for command input. Run the following command to launch Weston desktop.
[email protected]:/# weston --connector=29
If it fails to run, you can try the following commands with adb on an Ubuntu PC. Please make sure the Qualcomm Robotics RB5 development kit is connected to the Ubuntu PC with a USB cable. Please ignore this step if the above command runs successfully.
adb shellmkdir -p /usr/bin/weston_socketchmod 700 /usr/bin/weston_socketexport XDG_RUNTIME_DIR=/usr/bin/weston_socketexport LD_LIBRARY_PATH=/usr/lib:/usr/lib/aarch64-linux-gnu/weston --tty=1 --connector=29 --backend=drm-backend.so
4. Build OpenCV for Wayland
The Qualcomm Robotics RB5 development board supports Wayland/Weston as the display server, this OpenCV is a special version for Wayland compiled on the development board.
For More info about OpenCV refer:
https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html.Here are the steps to compile OpenCV on the target Qualcomm Robotics RB5 Development kit
[email protected]:/# apt-get install build-essential curl unzip[email protected]:/# apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev[email protected]:/# apt-get install libjpeg-dev libpng-dev[email protected]:/# apt-get install python-numpy libxkbcommon-dev libwayland-client0 libwayland-dev[email protected]:/# mkdir /home/src[email protected]:/# cd /home/src[email protected]:/# git clone https://github.com/pfpacket/opencv-wayland.git[email protected]:/# cd opencv-wayland/[email protected]:/# mkdir build[email protected]:/# cd build[email protected]:/# cmake -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/usr/local -DWITH_IPP=OFF -DWITH_WAYLAND=ON -DWITH_GTK=OFF ..[email protected]:/# make -j7
5. Build the TF lite library on the Qualcomm Robotics RB5 Development kit.
For more information see the link:https://www.tensorflow.org/lite/guide/build_arm64
First you need to install the TFlite dependencies using the following commands:
[email protected]:/# cd /home/src[email protected]:/# git clone https://github.com/tensorflow/tensorflow.git[email protected]:/# cd tensorflow[email protected]:/# git branch r2.1 remotes/origin/r2.1[email protected]:/# git checkout r2.1[email protected]:/#./tensorflow/lite/tools/make/download_dependencies.sh
Next step is to run the install script with following command. Script file name might be different depending on the version of TF lite that was cloned.
[email protected]:/# ./tensorflow/lite/tools/make/build_generic_aarch64_lib.sh
Please make sure the libtensorflow-lite.a is generated after the above steps are done. You will find the file at the following location:
cd home/src/tensorflow/tensorflow/lite/tools/make/gen/generic-aarch64_armv8-a/lib/
6. Build the object detection application on the Qualcomm Robotics RB5 Development kit.
The following steps are to build the application.
[email protected]:/# cd /home/src[email protected]:/# git clone https://github.com/mattn/webcam-detect-tflite.git[email protected]:/# cd webcam-detect-tflite
Copy the following content and save it as CMakeLists.txt to /home/src/webcam-detect-tflite
cmake_minimum_required(VERSION 3.0)project(webcam-detector)set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17 -O2 -g")set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ldl -lrt")message(STATUS "optional:-std=c++17")set(OpenCV_DIR "/home/src/opencv-wayland/build/")include_directories(/usr/include/freetype2/)include_directories(/home/src/tensorflow/)include_directories(/home/src/tensorflow/tensorflow/lite/tools/make/downloads/flatbuffers/include/)find_package(OpenCV REQUIRED)message(STATUS "OpenCV library status:")message(STATUS " version: ${OpenCV_VERSION}")message(STATUS " libraries: ${OpenCV_LIBS}")message(STATUS " include path: ${OpenCV_INCLUDE_DIRS}")include_directories( ${OpenCV_INCLUDE_DIRS} )ADD_LIBRARY(TFlite_LIB STATIC IMPORTED)SET_TARGET_PROPERTIES(TFlite_LIB PROPERTIESIMPORTED_LOCATION /home/src/tensorflow/tensorflow/lite/tools/make/gen/generic-aarch64_armv8-a/lib/libtensorflow-lite.a)set(SOURCE_FILESmain.cxx)add_executable(${PROJECT_NAME} ${SOURCE_FILES})target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} )target_link_libraries(${PROJECT_NAME}freetypepthreaddl)target_link_libraries(${PROJECT_NAME} TFlite_LIB)
After saving the above content to CMakeLists.txt, execute the following commands to build the sample app on target.
[email protected]:/# cd /home/src/webcam-detect-tflite[email protected]:/# cmake .[email protected]:/# make
If the above steps run successfully; the application webcam-detector will be generated under the folder build.
For the application to run, it will need a font file to display the objects on the screen. Please download the font file named "mplus-1c-thin.ttf" from this link and save it on the target device using the following command:
adb push mplus-1c-thin.ttf /home/src/webcam-detect-tflite
If you want to know more details about Tensorflow lite, please visit:
Usage Instructions
Before running the application, please check the USB camera node.
By default, the device node in kernel should be /dev/video2. You can reconnect the USB camera module to check if this node appears and disappears accordingly.
Run the following commands on the Qualcomm Robotics RB5 Development kit to run the application.
[email protected]:/# cd home/src/webcam-detect-tflite[email protected]:/# ./webcam-detector
Then you will see the classification result as shown below.

Contributors
Name | Title/Company |
---|---|
Weizhang Luo | Senior Staff Engineer, Qualcomm Communication Technologies (Shenzhen) Co., Ltd. |
Qualcomm Robotics RB5 and Qualcomm QRB6165 are products of Qualcomm Technologies, Inc. and/or its subsidiaries