Object detection with TensorFlow Lite

Skill LevelArea of FocusOperating SystemPlatform/Hardware
IntermediateComputer Vision, Embedded, Robotics, Sensors, Smart CitiesUbuntu / Ubuntu CoreQualcomm Robotics RBx Dev Kit
This project is designed to help you to use OpenCV to do objection detection on Qualcomm® Robotics RB5 development kit with a USB camera.

Objective

The main objective of this project is to start your artificial intelligence (AI) application development for robotics using the Qualcomm Robotics RB5 development kit. It will walk you through the following steps:

  1. Compile the TensorFlow lite libraries
  2. Test USB camera
  3. Run an objection detection application on the CPU of the Qualcomm Robotics RB5 development kit.

Materials Required / Parts List / Tools

Source Code / Source Examples / Application Executable

Build / Assembly Instructions

Hardware set up.

  • Connect the Qualcomm Robotics RB5 development kit to the monitor through HDMI cable.
  • Plugin a keyboard and a mouse to the development board.
  • Connect the USB camera module to the development board.

2. Wi-Fi setup on Qualcomm Robotics RB5 Development kit

Use the below command to setup the Wi-Fi connection.

root@qrb5165-rb5:/# vi /data/misc/wifi/wpa_supplicant.conf

Please modify the details listed below to set Wi-Fi host name and password,ten save it and reboot the device.

network={
ssid="YOUR_SSID_NAME"
key_mgmt=WPA-PSK
pairwise=TKIP CCMP
group=TKIP CCMP
psk="PASSWD"
}

After the reboot is finished, run the command “ifconfig” to check if the Wi-Fi is connected.

3. Run Weston desktop on Qualcomm Robotics RB5 Development kit

If the HDMI cable connects properly, after the device boot up, you will see the device boot up logs and then see a shell is ready for command input. Run the following command to launch Weston desktop.

root@qrb5165-rb5:/# Weston --connector=29

If it fails to run, you can try the following commands with adb on an Ubuntu PC. Please make sure the Qualcomm Robotics RB5 development kit is connected to the Ubuntu PC with a USB cable. Please ignore this step if the above command runs successfully.

adb shell
mkdir -p /usr/bin/weston_socket
chmod 700 /usr/bin/weston_socket
export XDG_RUNTIME_DIR=/usr/bin/weston_socket
export LD_LIBRARY_PATH=/usr/lib:/usr/lib/aarch64-linux-gnu/
weston --tty=1 --connector=29 --backend=drm-backend.so

4. Build the TF lite library on the Ubuntu host machine.

Here are the commands. For more information see the link:

https://www.tensorflow.org/lite/guide/build_arm64
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git branch r2.1 remotes/origin/r2.1
git checkout r2.1
./tensorflow/lite/tools/make/build_generic_aarch64_lib.sh

Please make sure the libtensorflow-lite.a is generated after the above steps are done.

5. Build OpenCV for Wayland

The Qualcomm Robotics RB5 development board supports Wayland/Weston as the display server, this OpenCV is a special version for Wayland compiled on the development board.

For More info about OpenCV refer:

https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html.

Here are the steps to compile OpenCV on the target Qualcomm Robotics RB5 Development kit

root@qrb5165-rb5:/# sudo apt-get install build-essential curl unzip
root@qrb5165-rb5:/# sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
root@qrb5165-rb5:/# sudo apt-get install libjpeg-dev libpng-dev
root@qrb5165-rb5:/# sudo apt-get install python-numpy libxkbcommon-dev libwayland-client0 libwayland-dev
root@qrb5165-rb5:/# mkdir /home/src
root@qrb5165-rb5:/# cd /home/src
root@qrb5165-rb5:/# git clone https://github.com/pfpacket/opencv-wayland.git
root@qrb5165-rb5:/# cd opencv-wayland/
root@qrb5165-rb5:/# mkdir build
root@qrb5165-rb5:/# cd build
root@qrb5165-rb5:/# cmake -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/usr/local -DWITH_IPP=OFF -DWITH_WAYLAND=ON -DWITH_GTK=OFF ..
root@qrb5165-rb5:/# make -j7

6. Build the object detection application on target

Qualcomm Robotics RB5 Development kit.

Before you build the application, install TFlite dependencies first.

Here are commands to install TFlite dependencies.

root@qrb5165-rb5:/# cd /home/src
root@qrb5165-rb5:/# git clone https://github.com/tensorflow/tensorflow.git
root@qrb5165-rb5:/# cd tensorflow
root@qrb5165-rb5:/# git branch r2.1 remotes/origin/r2.1
root@qrb5165-rb5:/# git checkout r2.1
root@qrb5165-rb5:/#./tensorflow/lite/tools/make/download_dependencies.sh

The following steps are to build the application.

root@qrb5165-rb5:/# cd /home/src
root@qrb5165-rb5:/# git clone https://github.com/mattn/webcam-detect-tflite.git
root@qrb5165-rb5:/# cd webcam-detect-tflite
root@qrb5165-rb5:/# mkdir libs

Place the libtensorflow-lite.a file generated on the step #4 to the libs folder using the command adb push libtensorflow-lite.a

Copy the following content and save it as CMakeList.txt to /home/src/webcam-detect-tflite

cmake_minimum_required(VERSION 3.0)
project(webcam-detector)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17 -O2 -g")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -ldl -lrt")
message(STATUS "optional:-std=c++17")
#set(OpenCV_DIR "/home/src/opencv-wayland/build/")
include_directories(/usr/include/freetype2/)
include_directories(/home/src/tensorflow/)
include_directories(/home/src/tensorflow/tensorflow/lite/tools/make/downloads/flatbuffers/include/)
find_package(OpenCV REQUIRED)
message(STATUS "OpenCV library status:")
message(STATUS " version: ${OpenCV_VERSION}")
message(STATUS " libraries: ${OpenCV_LIBS}")
message(STATUS " include path: ${OpenCV_INCLUDE_DIRS}")
include_directories( ${OpenCV_INCLUDE_DIRS} )
ADD_LIBRARY(TFlite_LIB STATIC IMPORTED)
SET_TARGET_PROPERTIES(TFlite_LIB PROPERTIES
IMPORTED_LOCATION /home/rb5/webcam-detect-tflite/libs/libtensorflow-lite.a)
set(SOURCE_FILES
main.cxx)
add_executable(${PROJECT_NAME} ${SOURCE_FILES})
target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} )
target_link_libraries(${PROJECT_NAME}
freetype
pthread
dl
)
target_link_libraries(${PROJECT_NAME} TFlite_LIB)

After saving the above content to CMakeList.txt, execute the following commands:

root@qrb5165-rb5:/# mkdir build
root@qrb5165-rb5:/# cd build
root@qrb5165-rb5:/# cmake ..
root@qrb5165-rb5:/# make

If the above steps run successfully; the application webcam-detector will be generated under the folder build.

If you want to know more details about Tensorflow lite, please visit:

https://www.tensorflow.org/lite

Usage Instructions

Before running the application, please check the USB camera node.

By default, the device node in kernel should be /dev/video2. You can reconnect the USB camera module to check if this node appears and disappears accordingly.

Run the following commands on the Qualcomm Robotics RB5 Development kit to run the application.

root@qrb5165-rb5:/# cd home/src/webcam-detect-tflite/build
root@qrb5165-rb5:/# ./webcam-detector

Then you will see the classification result as shown below.

the result image shown on screen

Contributors

NameTitle/Company
Weizhang LuoSenior Staff Engineer, Qualcomm Communication Technologies (Shenzhen) Co., Ltd.


Qualcomm Robotics RB5 and Qualcomm QRB6165 are products of Qualcomm Technologies, Inc. and/or its subsidiaries