Depth Mapping with Qualcomm® Robotics RB3 Dev Kit

• Robotics DragonBoard™ 845c development board - Based on the Qualcomm® SDA845 processor
• Power Adapter - 12 V with 2500 mA required by the 96Boards specification
• USB to Micro USB cable - For serial console interface and ADB, Fast boot commands
• USB to USB Type C cable - For connecting the USB3.0 Type C port and flashing images
• Ubuntu 14.04 LTS Linux distribution for 64-bit architecture - Host PC OS for Qualcomm Robotics RB3 SDK build process

Process for Creating Ecosystem for Depth Mapping in Qualcomm Robotics RB3 platform:

1. Download the Camera application source code in host machine for camera sample codes from Thundercomm website under ‘Sample_apps_codes’ (camera_test.tar)

2. Build and Transfer the application from host machine to Qualcomm Robotics RB3 as mentioned in the Qualcomm Robotics RB3 Linux User Guide

3. In order to capture a raw image with depth properties, operate the cameras in "Three camera concurrency modes" as discussed below.

4. Follow the instructions provided below to understand the commands in the program and then set the device’s four cameras in three different concurrency modes

• a) Build and execute the downloaded "camera_test" application on the host machine. This will generate “hal3_test” binary. Run the application ‘hal3_test’ with option ‘h’ to get the detailed instructions.“ The main purpose of this binary is to capture image streams provided by the cameras.

• b) This lists down the instructions to be prompted in a particular order to set the camera modes and also to trigger snapshots and videos

    / # hal3_test -h
    Enter Camera Testing
     ======================= Camera Test Version =======================
      CAMTEST_SHA1 :
      CAMTEST_BUILD_TS: 1/28/2019 11:38:6
      CAMTESTHOSTNAME : bogon
      CAMBUILD_IP : 10.0.36.26/24
     =====================================================================
    opt:h
    usage: hal3_test [-h] [-f command.txt]
    -h show usage
    -f using commands in file
    command in program:
    <order>:[Params]
     Orders:
     A: ADD a camera device for test
      >>A:id=0,psize=1920x1080,pformat=yuv420,ssize=1920x1080,sformat=jpeg
     U: Update meta setting
      >>U:manualaemode=1
     D: Delete current camera device
      >>D
     S: trigger Snapshot and dump preview/video
      >>S:2 take num(1,2,..) picture(s), eg. S:1
     s: trigger Snapshot
      >>s:2 take num(1,2,..) picture(s), eg. s:2
     v: triger video that switch from preview
      >>v:id=0,psize=1920x1080,pformat=yuv420,vsize=1920x1080,ssize=1920x1080,sformat=jpeg
     p: trigger dump Preview
      >>p:2 dump num(1,2,..) preview frame(s), eg. p:3
     M: set Metadata dump tag
      >>M:expvalue=1,scenemode=0
     Q: Quit

• c) As shown in the commands above, option ‘A’ will add a camera, which can be specified by mentioning ID. For ex: A:id=2 - ID 2 is specific to camera 2 to receive instructions

• d) Similarly, one can specify preview image size and format, also snapshot size and format through the options given against prompts

• e) Let us understand how to set the three camera concurrency mode to capture preview images with depth map

  • i. Run the camera application "hal3_test" on the device. Add CAM1(ToF), CAM2(Main Camera) and CAM3(Tracking), which is one the three concurrency combination with ToF being common to capture the depth.
  • ii. Give options to the prompts to add cameras and capture the preview with specified format and size and given in the below commands
    / # hal3_test
    Enter Camera Testing
     ======================= Camera Test Version =======================
     CAMTEST_SHA1 :
     CAMTEST_BUILD_TS: 1/28/2019 11:38:6
     CAMTESTHOSTNAME : bogon
     CAMBUILD_IP : 10.0.36.26/24
     =====================================================================
    Camera: 0 face:1
    Camera: 1 face:0
    Camera: 2 face:0
    Camera: 3 face:2
    CAM0>>A:id=2,psize=1920x1080,pformat=yuv420
    size: 0 Test camera:A:id=2,psize=1920x1080,pformat=yuv420
    camera:2
    preview size:1920x1080
    preview format:yuv420
    add a camera :2
    test mode:0[TESTMODE_TOF:3]
    preview:1920x1080 35
    preview:1920x1080 35
    CAM2>>P:1
    size: 0 Test camera:P:1
    preview request:1

• f) Add other two cameras from the current camera prompt as below and set the preview size and format one after the other

  CAM2>>A:id=3,psize=640x480,pformat=raw10
  size: 0 Test camera:A:id=3,psize=640x480,pformat=raw10
  camera:3
  preview size:640x480
  preview format:raw10
  add a camera :3
  test mode:0[TESTMODE_TOF:3]
  preview:640x480 37
  preview:640x480 37
  CAM3>>P:1
  size: 0 Test camera:P:1
  preview request:1
  CAM3>>A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
  size: 0 Test camera:A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
  camera:1
  preview size:640x480
  preview format:raw16
  raw size:640x480
  raw format:raw16
  add a camera :1
  test mode:3[TESTMODE_TOF:3]
  CameraId:1
  CAM1>>P:1
  size: 0 Test camera:P:1
  preview request:1

• g) Follow the below instructions to quit the application after deleting the cameras (D: Delete camera device, Q:Quit)

  CAM1>>D
  size: 0 Test camera:D
  CAM1>>Q
  size: 0 Test camera:Q
  quit
  Exiting application
  /#

5. Similarly, we can capture the depth images with other combinations with ToF, being common with the set of instructions in the same order as given below

• a) CAM1(ToF), CAM2(Main), CAM0(Stereo):

  # hal3_test
  >> A:id=2,psize=1920x1080,pformat=yuv420
  >> P:1
  >>A:id=0,psize=1280x720,pformat=yuv420,ssize=1280x720,sformat=jpeg,altek=1
  >> A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
  >> P:1
  >> D
  >> Q

• b) CAM1(ToF), CAM3(Tracking), CAM0(Stereo)

  # hal3_test
  >> A:id=3,psize=640x480,pformat=raw10
  >> P:1
  >>A:id=0,psize=1280x720,pformat=yuv420,ssize=1280x720,sformat=jpeg,altek=1
  >> A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16
  >> P:1
  >> D
  >> Q

The output folder for the images captured is "/data/misc/camera/".

6. The hardware for ToF has certain specifications and parameters for which the images captured are observed to have the following outcome

Camera can switch between two modes namely “Mode 0” and “Mode 1”

Following are the screenshots of sample depth map images captured using 3 camera concurrency modes

(images cannot be viewed using image viewers, a special tool called imageJ as mentioned below, is needed.)

These images can be viewed using “imageJ” tool. Download the tool from https://imagej.nih.gov/ij/ on the host machine.

With the limitations of depth sensing range varying from 0.2m to 4.5m, a depth image is captured in which the objects nearer are darker than those which are far.

Note:

• 1. Qualcomm Robotics RB3 ToF V4T module provides about 1% depth error and 2% depth variation (standard deviation).
• 2. The depth sensing ranges from 0.2 m to 4.5 m with '90-degree x 70-degree' wide FOV (Field of View), multiple signal output of depth, IR and background
• 3. Light scattering occurs due to unwanted reflections within the optics. This cannot be eliminated completely but can be avoided as much as possible by placing the device at a greatest distance possible from disturbing objects
• 4. Corners of depth map has large depth error due to small mismatch between lens FOV (Field of View) and diffuser FOI (Field of Illumination) of light source in the current TOF module