Elder Care with Qualcomm® Robotics RB3

Skill LevelArea of FocusOperating SystemPlatform/HardwareCloud Services/Platform
IntermediateComputer Vision, Healthcare, RoboticsLinuxRB3 Robotics Dev KitAmazon AWS IoT
This project is a proof of concept that demonstrates real-time object detection and pose estimation using an application from alwaysAI running on the Qualcomm Robotics RB3 platform, and an object detection ML model trained with AWS Sagemaker.
The project showcases the powerful combination of the Qualcomm Robotics RB3 platform, alwaysAI's deep learning computer vision developer platform, and the training capabilities of AWS Sagemaker. The alwaysAI computer vision development platform is designed to speed creation and deployment of machine learning applications on edge devices, such as Qualcomm Robotics RB3. The developer platform works with many AI model framework technologies and endpoint environments, and comes pre-loaded with a growing model catalog, starter apps, and core computer vision APIs (including object detection, tracking and counting; image classification, pose estimation, and semantic segmentation) enabling the development of a wide range of computer vision applications.Application Architecture for Elder Care Fall Detection Robot Built with AWS Services

In this project, the human is represented by a toy figure and the robot is detecting whether they are fine (standing) or in need of assistance (fallen down). The robot moves closer to the fallen person and could be extended to incorporate Alexa to be able to “speak” with the fallen person and take action accordingly.

  • The Robot Operating System (ROS) application, which encompasses alwaysAI computer vision application, running on Qualcomm Robotics RB3 Development Kit uses a local model, trained in AWS Sagemaker, to identify humans in video frames, develop situational awareness about the human, and offer assistance as needed.
  • The person detection model and the pose estimation models are pre-trained in AWS Sagemaker and downloaded to the Qualcomm Robotics RB3 platform.
  • The alwaysAI application is designed to run object detection and pose estimation on frames received from the camera. The results are used to decide any action needed and are displayed on a streamer service. ROS application reads the decision to move the robot and operates the wheels accordingly.
  • The robot detects human activity by analyzing the sequence of poses and classifies the human’s situation as safe or in need of help. In near future, the application will be extended to send a message to AWS IoT MQTT topic. The ROS application will integrate with Amazon Kinesis Video Streams ROS extension to capture live video feeds. We plan to use AWS Greengrass for deploying models on the robots. We also plan to integrate Alexa to initiate a conversation with the human under distress, call 911, etc.

Qualcomm Robotics RB3 is a product of Qualcomm Technologies, Inc. and/or its subsidiaries.

By submitting your content (“Submission”), you are granting Qualcomm a royalty-free, perpetual, non-exclusive, unrestricted, worldwide license to: (a) post, use, copy, sublicense, adapt, transmit, publicly perform or display any such Submission, (b) use, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, perform, play, host, communicate, make available and publish your Submission without restriction and (c) sublicense to third parties the unrestricted right to exercise any of the foregoing rights granted with respect to the Submission. The foregoing grants shall include the right to exploit any ideas, concepts, intellectual property, or proprietary rights in such Submission, including but not limited to rights under copyright, trademark, servicemark or patent laws under any relevant jurisdiction without Qualcomm owing any monies to you whatsoever. You represent and warrant that you own all right, title and interest in and to the Submission, or you have been granted sufficient rights in and to the Submission allowing the foregoing use of such Submission.