Snapdragon and Qualcomm branded products are products of
Qualcomm Technologies, Inc. and/or its subsidiaries.
Recently, we announced the new Qualcomm® Robotics RB5 Platform, a next-generation, robotics solution that can be used to develop high-compute, artificial intelligence (AI)-enabled, low-power robots and drones for consumer, enterprise, defense, industrial, and professional service applications.
Built around the Qualcomm® QRB5165 processor, the Qualcomm Robotics RB5 platform includes many of the features found in our cutting-edge Snapdragon® 865 Mobile Platform including its Qualcomm® Hexagon™ DSP with specialized compute capabilities for AI, and 5G connectivity. This makes the platform well suited for adding AI operations such as on-device intelligence and computer vision algorithms to your robotic and drone projects.
To help you get started, you can use the Qualcomm Robotics RB5 Development Kit along with the following learning resources:
- Quick Start Guide
- Hardware Reference Guide
- Software Reference Manual
- Example projects
- Developer forum
In this blog, we'll start by talking about a few of the unique features of the Qualcomm Robotics RB5 platform. We'll then focus on key aspects of the projects we recently posted, which demonstrate how to integrate the platform into your development.
Notable Features of the Qualcomm Robotics RB5 Platform
Strong wireless communications are key for controlling and receiving data from today's robots. One feature that makes the Qualcomm Robotics RB5 platform cutting edge is its support for 4G and 5G connectivity including support for mmWave. With 5G infrastructure now taking root, robotics developers can choose to use the optional 5G Mezzanine board to take advantage of 5G's higher speeds and lower latencies. In addition, the Qualcomm Robotics RB5 platform also supports Bluetooth 5.1 and various Wi-Fi modes including Wi-Fi 6. These features provide developers with a rich set of wireless connectivity solutions, both for communicating with the robot, and for communications between the Qualcomm Robotics RB5 platform and other wireless components that may be on the robot.
Another notable feature is the platform's support for computer vision tasks such as object detection. This is made possible by the platform's ability to process 4K or 8K HDR video and 200 megapixel photos, while handling up to seven cameras simultaneously. The platform has also been designed to handle operating environment temperatures of -40°C to 105°C.
At the operating system (OS) level, the platform is unique from our other hardware offerings in that it runs on Ubuntu Linux, Yocto, and Robot Operating System (ROS) 2.0 instead of the usual Android OS. These operating systems provide a much more robotics-focused environment for the platform's intended purpose.
Software wise, developers can take advantage of the Hexagon DSP SDK, Qualcomm® Computer Vision SDK, and Qualcomm® Neural Processing SDK for AI for computer vision and other AI objectives. The platform also uses a port of the Android Neural Networks API (NNAPI) which has been designed to run on the platform's Hexagon DSP. NNAPI provides the basis for running neural network inference on machine learning (ML) models developed in frameworks like TensorFlow.
In addition to the support for above mentioned SDK's, the platform provides GST plug-in(s) that can be called by the application developer to exercise various subsystems. For example, Qualcomm Technologies MMF SRC GStreamer element can be used to abstract out the camera and video multimedia architecture for capture and encoding. This can act as a source to the GStreamer-SNPE and TF Lite plugins which are available to simplify and optimize AI and ML workloads. These plugins have direct bindings with the HTA and can be configured to run on DSP, CPU, or GPU.
The use of GStreamer and support for ROS facilitates chaining multiple components together. For example, the output of a source element can easily be consumed by a ROS node. The ROS node can then act as a consumer on any user defined topic. A ROS node can also be a producer for a GStreamer sink element. This allows the platform to behave in a distributed manner.
Projects to Help you Get Started with Your Development
To see how some of the aforementioned capabilities of the Qualcomm Robotics RB5 platform work, the following example projects are available on QDN. Below we've highlighted some of the notable aspects on how they can help you get started with your development on the platform.
Auto exploration with navigation
This project shows how autonomous navigation could be achieved using a TurtleBot3 Waffle Pi navigable robot connected to the Qualcomm Robotics RB5 platform. In this example, the robot explores its surrounding area until it locates all boundaries.
Figure 1 - Image of the TurtleBot3 Waffle Pi. The project includes some basic instructions for assembly and connecting the Qualcomm Robotics RB5 Development Kit to the TurtleBot3's OpenCR controller board over USB. On the software side, steps are included for installing ROS and navigation packages onto the robot, and how to SSH into the RB5. The RB5 is then used to execute a self-navigation program that runs on the OpenCR controller.
To help developers better understand the robot's movements, the project also makes use of rviz, a 3D visualization tool for ROS. rviz displays a rich set of data from the robot including its sensors and cameras, which can be especially useful during debugging versus trying to correlate data in debugger windows.
Figure 2 - Screenshot of rviz showing both the camera stream and boundary mapping performed by the robot. Object detection with TensorFlow Lite
TensorFlow is one of the most popular ML frameworks at the moment. And with the rise of ML inference at the edge, TensorFlow Lite offers mobile developers a mobile-optimized version of TensorFlow with a smaller binary footprint.
The Object detection with TensorFlow Lite project demonstrates how to hook up a USB camera to the Qualcomm Robotics RB5 Development Kit and use it in conjunction with TensorFlow Lite for object detection. To facilitate this functionality, the demo makes use of an open-source example project on GitHub written in C.
The steps of the project cover how to set up Wi-Fi on the Qualcomm Robotics RB5 Development Kit, how to build and install TensorFlow Lite, and how to build OpenCV for it.
The project also makes use of the Wayland display server which provides a desktop environment on the platform that is displayed on a monitor connected to the development kit's HDMI port. Through this desktop, developers can work directly on the dev kit and even view the incoming camera stream.
Robotic limbs that can move and manipulate objects, are probably one of the first images that come to mind when envisioning a traditional robot.
The OpenManipulator with MoveIt! project demonstrates how to control the OpenManipulator robotic arm via the Qualcomm Robotics RB5 Development Kit in conjunction with an OpenCR controller.
The project shows how to install ROS and the MoveIt robotics manipulation framework on the Qualcomm Robotics RB5 Development Kit. It also shows the execution of a Python app running on the platform that controls the arm through high-level commands which set the joint positions of the arms.
The project also demonstrates how rviz can be used to visualize the arm and its movements.
Qualcomm Robotics RB5 Development Kit with Alexa skills
Smart speakers have become a popular method for interacting with devices. This project shows how voice commands sent via an Amazon Echo, could be used to control a robot. While the project doesn't directly control a robot, it does demonstrate how to use Amazon Skills in conjunction with an event listener implemented on the Qualcomm Robotics RB5 Development Kit as an ROS node. The event handler is registered with an AWS Lambda function and shows how to translate an incoming message into robotic action commands (e.g., move, change speed, stop).
The project's architecture is shown here:
Figure 3 – Project architecture for integrating the platform with Alexa Skills.
Conclusion
The Qualcomm Robotics RB5 platform is truly a next-generation robotics platform, that brings the advanced computing of the Qualcomm QRB5165 processor along with 5G connectivity, to robotic and drone projects. Moreover, it adds advanced capabilities such as object detection that can be used in conjunction with robotic controllers (e.g., OpenCR) to build smart and autonomous robots and drones.
Given all of the functionality packed into the Qualcomm Robotics RB5 platform, it's no wonder that Darin Andersen, chief executive officer, cofounder, NXT Robotics Corp, said: "The new Qualcomm Robotics RB5 platform from Qualcomm Technologies is a dream chipset come true".
We hope the projects listed above will help you to get started with the Qualcomm Robotics RB5 platform. And be sure to revisit our projects page frequently, as we are adding new projects all the time. You can easily filter on robotic projects by select "Robotics" under the "Area of Focus" menu on the left side of that page.
If you have a robotics project built with Qualcomm Technologies solutions that you would like to show off, be sure to submit it here.
Qualcomm Robotics, Snapdragon, Qualcomm Hexagon, Qualcomm Computer Vision, and Qualcomm Neural Processing SDK, are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
