Object detection with TensorFlow Lite

Skill LevelArea of FocusOperating SystemPlatform/Hardware
IntermediateComputer Vision, Embedded, Robotics, Sensors, Smart CitiesUbuntu / Ubuntu CoreQualcomm Robotics RBx Dev Kit
This project is designed to help you to use OpenCV to do objection detection on Qualcomm® Robotics RB5 development kit with a USB camera.

Objective

The main objective of this project is to start your artificial intelligence (AI) application development for robotics using the Qualcomm Robotics RB5 development kit. It will walk you through the following steps:

  1. Compile the TensorFlow lite libraries
  2. Test USB camera
  3. Run an objection detection application on the CPU of the Qualcomm Robotics RB5 development kit.

Materials Required / Parts List / Tools

Source Code / Source Examples / Application Executable

Build / Assembly Instructions

Hardware set up.

  • Connect the Qualcomm Robotics RB5 development kit to the monitor through HDMI cable.
  • Plugin a keyboard and a mouse to the development board.
  • Connect the USB camera module to the development board.

2. Wi-Fi setup on Qualcomm Robotics RB5 Development kit

Use the below command to setup the Wi-Fi connection.

root@qrb5165-rb5:/# vi /data/misc/wifi/wpa_supplicant.conf

Please modify the details listed below to set Wi-Fi host name and password, then save it and reboot the device.

network={
ssid="YOUR_SSID_NAME"
key_mgmt=WPA-PSK
pairwise=TKIP CCMP
group=TKIP CCMP
psk="PASSWD"
}

After the reboot is finished, run the command “ifconfig” to check if the Wi-Fi is connected.

3. Run Weston desktop on Qualcomm Robotics RB5 Development kit

If the HDMI cable connects properly, after the device boot up, you will see the device boot up logs and then see a shell is ready for command input. Run the following command to launch Weston desktop.

root@qrb5165-rb5:/# Weston --connector=29

If it fails to run, you can try the following commands with adb on an Ubuntu PC. Please make sure the Qualcomm Robotics RB5 development kit is connected to the Ubuntu PC with a USB cable. Please ignore this step if the above command runs successfully.

adb shell
mkdir -p /usr/bin/weston_socket
chmod 700 /usr/bin/weston_socket
export XDG_RUNTIME_DIR=/usr/bin/weston_socket
export LD_LIBRARY_PATH=/usr/lib:/usr/lib/aarch64-linux-gnu/
weston --tty=1 --connector=29 --backend=drm-backend.so

4. Build OpenCV for Wayland

The Qualcomm Robotics RB5 development board supports Wayland/Weston as the display server, this OpenCV is a special version for Wayland compiled on the development board.

For More info about OpenCV refer:

https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html.

Here are the steps to compile OpenCV on the target Qualcomm Robotics RB5 Development kit

root@qrb5165-rb5:/# apt-get install build-essential curl unzip
root@qrb5165-rb5:/# apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
root@qrb5165-rb5:/# apt-get install libjpeg-dev libpng-dev
root@qrb5165-rb5:/# apt-get install python-numpy libxkbcommon-dev libwayland-client0 libwayland-dev
root@qrb5165-rb5:/# mkdir /home/src
root@qrb5165-rb5:/# cd /home/src
root@qrb5165-rb5:/# git clone https://github.com/pfpacket/opencv-wayland.git
root@qrb5165-rb5:/# cd opencv-wayland/
root@qrb5165-rb5:/# mkdir build
root@qrb5165-rb5:/# cd build
root@qrb5165-rb5:/# cmake -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/usr/local -DWITH_IPP=OFF -DWITH_WAYLAND=ON -DWITH_GTK=OFF ..
root@qrb5165-rb5:/# make -j7

5. Build the TF lite library on the Qualcomm Robotics RB5 Development kit.

For more information see the link:https://www.tensorflow.org/lite/guide/build_arm64

First you need to install the TFlite dependencies using the following commands:

root@qrb5165-rb5:/# cd /home/src
root@qrb5165-rb5:/# git clone https://github.com/tensorflow/tensorflow.git
root@qrb5165-rb5:/# cd tensorflow
root@qrb5165-rb5:/# git branch r2.1 remotes/origin/r2.1
root@qrb5165-rb5:/# git checkout r2.1
root@qrb5165-rb5:/#./tensorflow/lite/tools/make/download_dependencies.sh

Next step is to run the install script with following command. Script file name might be different depending on the version of TF lite that was cloned.

root@qrb5165-rb5:/# ./tensorflow/lite/tools/make/build_generic_aarch64_lib.sh

Please make sure the libtensorflow-lite.a is generated after the above steps are done. You will find the file at the following location:

cd home/src/tensorflow/tensorflow/lite/tools/make/gen/aarch64_armv8-a/lib/

6. Build the object detection application on the Qualcomm Robotics RB5 Development kit.

The following steps are to build the application.

root@qrb5165-rb5:/# cd /home/src
root@qrb5165-rb5:/# git clone https://github.com/mattn/webcam-detect-tflite.git
root@qrb5165-rb5:/# cd webcam-detect-tflite
root@qrb5165-rb5:/# mkdir libs

Copy the following content and save it as CMakeLists.txt to /home/src/webcam-detect-tflite

cmake_minimum_required(VERSION 3.0)
project(webcam-detector)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17 -O2 -g")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ldl -lrt")
message(STATUS "optional:-std=c++17")
#set(OpenCV_DIR "/home/src/opencv-wayland/build/")
include_directories(/usr/include/freetype2/)
include_directories(/home/src/tensorflow/)
include_directories(/home/src/tensorflow/tensorflow/lite/tools/make/downloads/flatbuffers/include/)
find_package(OpenCV REQUIRED)
message(STATUS "OpenCV library status:")
message(STATUS " version: ${OpenCV_VERSION}")
message(STATUS " libraries: ${OpenCV_LIBS}")
message(STATUS " include path: ${OpenCV_INCLUDE_DIRS}")
include_directories( ${OpenCV_INCLUDE_DIRS} )
ADD_LIBRARY(TFlite_LIB STATIC IMPORTED)
SET_TARGET_PROPERTIES(TFlite_LIB PROPERTIES
IMPORTED_LOCATION /home/src/tensorflow/tensorflow/lite/tools/make/gen/generic-aarch64_armv8-a/lib/libtensorflow-lite.a)
set(SOURCE_FILES
main.cxx)
add_executable(${PROJECT_NAME} ${SOURCE_FILES})
target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} )
target_link_libraries(${PROJECT_NAME}
freetype
pthread
dl
)
target_link_libraries(${PROJECT_NAME} TFlite_LIB)

After saving the above content to CMakeLists.txt, execute the following commands to build the sample app on target.

root@qrb5165-rb5:/# cd /home/src/webcam-detect-tflite
root@qrb5165-rb5:/# cmake .
root@qrb5165-rb5:/# make

If the above steps run successfully; the application webcam-detector will be generated under the folder build.

For the application to run, it will need a font file to display the objects on the screen. Please download the font file named "mplus-1c-thin.ttf" from this link and save it on the target device using the following command:

adb push mplus-1c-thin.ttf /home/src/webcam-detect-tflite

If you want to know more details about Tensorflow lite, please visit:

https://www.tensorflow.org/lite

Usage Instructions

Before running the application, please check the USB camera node.

By default, the device node in kernel should be /dev/video2. You can reconnect the USB camera module to check if this node appears and disappears accordingly.

Run the following commands on the Qualcomm Robotics RB5 Development kit to run the application.

root@qrb5165-rb5:/# cd home/src/webcam-detect-tflite
root@qrb5165-rb5:/# ./webcam-detector

Then you will see the classification result as shown below.

the result image shown on screen

Contributors

NameTitle/Company
Weizhang LuoSenior Staff Engineer, Qualcomm Communication Technologies (Shenzhen) Co., Ltd.


Qualcomm Robotics RB5 and Qualcomm QRB6165 are products of Qualcomm Technologies, Inc. and/or its subsidiaries