Depth Mapping with Qualcomm® Robotics RB3 Dev Kit

Skill LevelArea of FocusOperating SystemPlatform/Hardware
BeginnerComputer Vision, Embedded, Robotics, SensorsLinuxQualcomm Robotics RB3 Robotics Dev Kit, Artificial Intelligence
A wide variety of robotics use cases need Depth Mapping for the robots to navigate. This project is designed to help you get started with depth mapping on the Qualcomm Robotics RB3 Development Kit. The project will show how you can capture a raw image with depth map and use it for future use cases such as background blurring etc.
To create depth mapping ecosystem using Qualcomm Robotics RB3 platform for use-case development.
  • Robotics DragonBoard™ 845c development board - Based on the Qualcomm® SDA845 processor
  • Power Adapter - 12 V with 2500 mA required by the 96Boards specification
  • USB to Micro USB cable - For serial console interface and ADB, Fast boot commands
  • USB to USB Type C cable - For connecting the USB3.0 Type C port and flashing images
  • Ubuntu 14.04 LTS Linux distribution for 64-bit architecture - Host PC OS for Qualcomm Robotics RB3 SDK build process

camera_test.tar

(found on the Thundercomm Qualcomm Robotics RB3 Development Kit web page, under the Technical Documents tab in the “Sample-apps-codes” section)

Process for Creating Ecosystem for Depth Mapping in Qualcomm Robotics RB3 platform:

1. Download the Camera application source code in host machine for camera sample codes from Thundercomm website under ‘Sample_apps_codes’ (camera_test.tar)

2. Build and Transfer the application from host machine to Qualcomm Robotics RB3 as mentioned in the Qualcomm Robotics RB3 Linux User Guide

3. In order to capture a raw image with depth properties, operate the cameras in "Three camera concurrency modes" as discussed below.

4. Follow the instructions provided below to understand the commands in the program and then set the device’s four cameras in three different concurrency modes

  • a) Build and execute the downloaded "camera_test" application on the host machine. This will generate “hal3_test” binary. Run the application ‘hal3_test’ with option ‘h’ to get the detailed instructions.“ The main purpose of this binary is to capture image streams provided by the cameras.

  • b) This lists down the instructions to be prompted in a particular order to set the camera modes and also to trigger snapshots and videos

      / # hal3_test -h
      Enter Camera Testing
       ======================= Camera Test Version =======================
        CAMTEST_SHA1 :
        CAMTEST_BUILD_TS: 1/28/2019 11:38:6
        CAMTESTHOSTNAME : bogon
        CAMBUILD_IP : 10.0.36.26/24
       =====================================================================
      opt:h
      usage: hal3_test [-h] [-f command.txt]
      -h show usage
      -f using commands in file
      command in program:
      <order>:[Params]
       Orders:
       A: ADD a camera device for test
        >>A:id=0,psize=1920x1080,pformat=yuv420,ssize=1920x1080,sformat=jpeg
       U: Update meta setting
        >>U:manualaemode=1
       D: Delete current camera device
        >>D
       S: trigger Snapshot and dump preview/video
        >>S:2 take num(1,2,..) picture(s), eg. S:1
       s: trigger Snapshot
        >>s:2 take num(1,2,..) picture(s), eg. s:2
       v: triger video that switch from preview
        >>v:id=0,psize=1920x1080,pformat=yuv420,vsize=1920x1080,ssize=1920x1080,sformat=jpeg
       p: trigger dump Preview
        >>p:2 dump num(1,2,..) preview frame(s), eg. p:3
       M: set Metadata dump tag
        >>M:expvalue=1,scenemode=0
       Q: Quit
  • c) As shown in the commands above, option ‘A’ will add a camera, which can be specified by mentioning ID. For ex: A:id=2 - ID 2 is specific to camera 2 to receive instructions

  • d) Similarly, one can specify preview image size and format, also snapshot size and format through the options given against prompts

  • e) Let us understand how to set the three camera concurrency mode to capture preview images with depth map

    • i. Run the camera application "hal3_test" on the device. Add CAM1(ToF), CAM2(Main Camera) and CAM3(Tracking), which is one the three concurrency combination with ToF being common to capture the depth.
    • ii. Give options to the prompts to add cameras and capture the preview with specified format and size and given in the below commands
      / # hal3_test
      Enter Camera Testing
       ======================= Camera Test Version =======================
       CAMTEST_SHA1 :
       CAMTEST_BUILD_TS: 1/28/2019 11:38:6
       CAMTESTHOSTNAME : bogon
       CAMBUILD_IP : 10.0.36.26/24
       =====================================================================
      Camera: 0 face:1
      Camera: 1 face:0
      Camera: 2 face:0
      Camera: 3 face:2
      CAM0>>A:id=2,psize=1920x1080,pformat=yuv420
      size: 0 Test camera:A:id=2,psize=1920x1080,pformat=yuv420
      camera:2
      preview size:1920x1080
      preview format:yuv420
      add a camera :2
      test mode:0[TESTMODE_TOF:3]
      preview:1920x1080 35
      preview:1920x1080 35
      CAM2>>P:1
      size: 0 Test camera:P:1
      preview request:1
  • f) Add other two cameras from the current camera prompt as below and set the preview size and format one after the other

    CAM2>>A:id=3,psize=640x480,pformat=raw10
    size: 0 Test camera:A:id=3,psize=640x480,pformat=raw10
    camera:3
    preview size:640x480
    preview format:raw10
    add a camera :3
    test mode:0[TESTMODE_TOF:3]
    preview:640x480 37
    preview:640x480 37
    CAM3>>P:1
    size: 0 Test camera:P:1
    preview request:1
    CAM3>>A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
    size: 0 Test camera:A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
    camera:1
    preview size:640x480
    preview format:raw16
    raw size:640x480
    raw format:raw16
    add a camera :1
    test mode:3[TESTMODE_TOF:3]
    CameraId:1
    CAM1>>P:1
    size: 0 Test camera:P:1
    preview request:1
  • g) Follow the below instructions to quit the application after deleting the cameras (D: Delete camera device, Q:Quit)

    CAM1>>D
    size: 0 Test camera:D
    CAM1>>Q
    size: 0 Test camera:Q
    quit
    Exiting application
    /#

5. Similarly, we can capture the depth images with other combinations with ToF, being common with the set of instructions in the same order as given below

  • a) CAM1(ToF), CAM2(Main), CAM0(Stereo):

    # hal3_test
    >> A:id=2,psize=1920x1080,pformat=yuv420
    >> P:1
    >>A:id=0,psize=1280x720,pformat=yuv420,ssize=1280x720,sformat=jpeg,altek=1
    >> A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
    >> P:1
    >> D
    >> Q
  • b) CAM1(ToF), CAM3(Tracking), CAM0(Stereo)

    # hal3_test
    >> A:id=3,psize=640x480,pformat=raw10
    >> P:1
    >>A:id=0,psize=1280x720,pformat=yuv420,ssize=1280x720,sformat=jpeg,altek=1
    >> A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
    >> P:1
    >> D
    >> Q

The output folder for the images captured is "/data/misc/camera/".

6. The hardware for ToF has certain specifications and parameters for which the images captured are observed to have the following outcome

Camera can switch between two modes namely “Mode 0” and “Mode 1”

PerformanceMode 0: Short rangeMode 1: Long range
Camera Distance0.2 to 1.2 [m]0.5 to 4.5 [m]
Frame Rate30 fps20 fps

Following are the screenshots of sample depth map images captured using 3 camera concurrency modes

(images cannot be viewed using image viewers, a special tool called imageJ as mentioned below, is needed.)

These images can be viewed using “imageJ” tool. Download the tool from https://imagej.nih.gov/ij/ on the host machine.

With the limitations of depth sensing range varying from 0.2m to 4.5m, a depth image is captured in which the objects nearer are darker than those which are far.

Note:

  • 1. Qualcomm Robotics RB3 ToF V4T module provides about 1% depth error and 2% depth variation (standard deviation).
  • 2. The depth sensing ranges from 0.2 m to 4.5 m with '90-degree x 70-degree' wide FOV (Field of View), multiple signal output of depth, IR and background
  • 3. Light scattering occurs due to unwanted reflections within the optics. This cannot be eliminated completely but can be avoided as much as possible by placing the device at a greatest distance possible from disturbing objects
  • 4. Corners of depth map has large depth error due to small mismatch between lens FOV (Field of View) and diffuser FOI (Field of Illumination) of light source in the current TOF module
NameTitle/Company
Rakesh SankarSr, System Architect
Global Edge Software Ltd
Ashish Tiwari
Vihari SreeramSenior Software Engineer
Global Edge Software Ltd
Akshay KulkarniTechnical Lead
Global Edge Software Ltd

By submitting your content (“Submission”), you are granting Qualcomm a royalty-free, perpetual, non-exclusive, unrestricted, worldwide license to: (a) post, use, copy, sublicense, adapt, transmit, publicly perform or display any such Submission, (b) use, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, perform, play, host, communicate, make available and publish your Submission without restriction and (c) sublicense to third parties the unrestricted right to exercise any of the foregoing rights granted with respect to the Submission. The foregoing grants shall include the right to exploit any ideas, concepts, intellectual property, or proprietary rights in such Submission, including but not limited to rights under copyright, trademark, service mark or patent laws under any relevant jurisdiction without Qualcomm owing any monies to you whatsoever. You represent and warrant that you own all right, title and interest in and to the Submission, or you have been granted sufficient rights in and to the Submission allowing the foregoing use of such Submission.