A Virtual Showcase of the Open-source Development Model

For Linux developers who build low-level kernel and driver solutions, the open-source development model plays a key role in helping to drive innovation forward. This model allows you to take advantage of the features and ongoing updates contributed by the Linux development community, as well as make your own contributions. Today, many organizations embrace the open-source development model and the benefits it brings.

Recently, we teamed up with Linaro (the organization that oversees open source software for ARM instruction sets) to present Linaro Tech Day 2022. This virtual event showcased how we work with Linaro open-source initiatives, and how open source plays a pivotal role in our portfolio of technologies.

Let’s take a brief look at a couple of the presentations that illustrate this.

Robotics in IoT and Upstreaming
Ameya Pandit, Sr. Staff Engineer at Qualcomm Innovation Center (QuIC), presented Robotics in IoT and Upstreaming. He covered how the team behind the Qualcomm Robotics RB5 Development Kit upstreams Linux enhancements and fixes for the platform’s Qualcomm QRB5165 processor and keeps them in sync with the mainline Linux kernel as it evolves (see below). The team provides software evaluation builds based on the Yocto project and recent Linux kernel versions, while all non-HLOS (high-level OS) binaries are derived from our release infrastructure.

Overview of the release process for the Qualcomm Robotics RB5.

The mainline kernel from kernel.org (the main distribution point for the Linux kernel’s source code) keeps evolving and growing. The team takes a version of this mainline kernel to make their enhancements and create a the downstream version of the kernel, after which it goes through an internal review before being shared with the developer community through Code Linaro. At same time, it’s also released to external OEMs. Finally, Linaro integrates (upstreams) the updated kernel back into the mainline so that the community and partners can benefit from the team’s enhancements and fixes. Ameya briefly reviewed some of these enhancements that are upstreamed for the QRB5165 SOC.

Ameya then showed three demos of the Qualcomm Robotics RB5 in action. The first demo, depicted below, shows how the Qualcomm Robotics RB5 running ROS can be controlled using voice-activated Alexa commands:

Left: Alexa device to capture voice commands; Right: Qualcomm Robotics RB5 responding to the voice commands – in this example, increasing its movement speed across the floor.

For more information, you can find the demo here amongst our Projects pages to try it yourself.

The second demo showed how the Qualcomm Robotics RB5 can perform gesture recognition and use the resulting pose information to manipulate a robotic arm accordingly.

In this demo, the camera stream is fed to the Qualcomm Robotics RB5 running ROS which runs inference to perform the gesture recognition. It then sends the corresponding movement commands to a microcontroller unit (MCU) attached to an OpenManipulator robotic arm over CAN bus to perform the final actuator movements.

In this pipeline, the live stream is captured by the camera. The stream is fed through a gesture recognition algorithm powered by TensorFlow Lite to generate pose metadata. The data is also used to render a skeletal overlay on a display screen to show how the captured pose corresponds to the operator’s real pose. Finally, the pose data is used to generate movement commands which are sent to the arm.

The final demo in Ameya’s presentation showed how the Qualcomm Robotics RB5 can be used as the basis for a robot that can find, identify, and grab objects, in this case, a paper cup.

Robot powered by the Qualcomm Robotics RB5 using image recognition to identify and pick up an object.

For this demo, an OpenManipulator was mounted onto a TurtleBot3, and equipped with an Intel RealSense D435i camera for depth sensing. These components were then integrated with the Qualcomm Robotics RB5 platform via USB connections.

Communication between the Qualcomm Robotics RB5 and various components is performed over USB.

During operation, the camera streams the robot’s view as it navigates, and the Qualcomm Robotics RB5 runs object detection to look for plastic cups in each frame. Once identified, the unit is commanded to move into position while the arm is instructed to grab the cup. The arm movements are controlled using the DYNAMIXEL SDK. See below for the flow of information that makes this possible:

Flow of data from image recognition to actuator movements

Here, the camera stream is analyzed by a computer vision algorithm (represented by the Camera Node) which uses an RCNN to identify if a cup was found. Once identified, coordinates to reach and pick up the cup are computed, and subsequent commands are sent from the Qualcomm Robotics RB5 to the OpenManipulator and TurtleBot3 to carry out the appropriate actuator movements.

Qualcomm Cloud AI 100: Production Upstream Linux

Jeff Hugo, Staff Engineer, Qualcomm Technologies, Inc. presented: Qualcomm Cloud AI 100: Production Upstream Linux.

Jeff began with an overview of the Qualcomm Cloud AI 100, a PCIe-based card that runs cloud-based inference such as natural language processing, image classification, and recommendation systems. As a niche product, it’s discretely made available to customers who want to run ML workloads across different regions, edge deployments, and datacenters, with the option to run multiple cards in parallel.

Jeff then described how his team produces the Direct Rendering Manager (DRM) accelerator driver for the Linux kernel and the Modern Host Interface (MHI) stack used on the Cloud AI 100. These offerings support x86 and Arm64 and Linux 3.10-5.16 across multiple distros (CentOS, Red Hat Enterprise Linux, Ubuntu, etc.). Currently Qualcomm Cloud AI consists of about 10k lines of code across 14 files and 300 commits, and the driver has been implemented as a miscellaneous device. It makes use of several kernel facilities including PCI, DMA-buf, HWMON, sysfs, debugfs, and the DMA API.

The release procedure for the driver makes use of Dynamic Kernel Module Support on GitHub. The team deploys it as source code to customers, who in turn, compile the driver on their deployed systems utilizing back-port logic to work with their existing systems.

On the MHI side, the code is upstreamed and maintained by Linaro. It currently has 24 unique commit authors as of v5.19-rc4, including two from Linaro and five from QTI.

Experience the Benefits of Open Source Development
As these demos have shown, open-source development plays a pivotal role in our portfolio of technologies. We not only benefit from the great contributions by the Linux development community, but also give back through the fixes and enhancements we make during the development of our technologies.

Interested in learning more?

• Check out the Linaro Resources Hub where you can watch all of the presentations from Linaro Tech Day 2022.
• Then, be sure to check out our Projects page on QDN where you can see more of our solutions in action and try them yourself.

Qualcomm Robotics RB5, Qualcomm QRB5165, and Qualcomm Cloud AI are products of Qualcomm Technologies, Inc. and/or its subsidiaries.