Skill Level | Area of Focus | Operating System | Platform/Hardware |
---|---|---|---|
Beginner | Computer Vision, Embedded, Robotics, Sensors | Linux | Qualcomm Robotics RB3 Robotics Dev Kit, Artificial Intelligence |
Objective
Materials Required / Parts List / Tools
- Robotics DragonBoard™ 845c development board - Based on the Qualcomm® SDA845 processor
- Power Adapter - 12 V with 2500 mA required by the 96Boards specification
- USB to Micro USB cable - For serial console interface and ADB, Fast boot commands
- USB to USB Type C cable - For connecting the USB3.0 Type C port and flashing images
- Ubuntu 14.04 LTS Linux distribution for 64-bit architecture - Host PC OS for Qualcomm Robotics RB3 SDK build process
Source code: Camera Application
camera_test.tar
(found on the Thundercomm Qualcomm Robotics RB3 Development Kit web page, under the Technical Documents tab in the “Sample-apps-codes” section)
Build / Assembly Instructions
Process for Creating Ecosystem for Depth Mapping in Qualcomm Robotics RB3 platform:
1. Download the Camera application source code in host machine for camera sample codes from Thundercomm website under ‘Sample_apps_codes’ (camera_test.tar)
2. Build and Transfer the application from host machine to Qualcomm Robotics RB3 as mentioned in the Qualcomm Robotics RB3 Linux User Guide
3. In order to capture a raw image with depth properties, operate the cameras in "Three camera concurrency modes" as discussed below.
4. Follow the instructions provided below to understand the commands in the program and then set the device’s four cameras in three different concurrency modes
a) Build and execute the downloaded "camera_test" application on the host machine. This will generate “hal3_test” binary. Run the application ‘hal3_test’ with option ‘h’ to get the detailed instructions.“ The main purpose of this binary is to capture image streams provided by the cameras.
b) This lists down the instructions to be prompted in a particular order to set the camera modes and also to trigger snapshots and videos
/ # hal3_test -hEnter Camera Testing======================= Camera Test Version =======================CAMTEST_SHA1 :CAMTEST_BUILD_TS: 1/28/2019 11:38:6CAMTESTHOSTNAME : bogonCAMBUILD_IP : 10.0.36.26/24=====================================================================opt:husage: hal3_test [-h] [-f command.txt]-h show usage-f using commands in filecommand in program:<order>:[Params]Orders:A: ADD a camera device for test>>A:id=0,psize=1920x1080,pformat=yuv420,ssize=1920x1080,sformat=jpegU: Update meta setting>>U:manualaemode=1D: Delete current camera device>>DS: trigger Snapshot and dump preview/video>>S:2 take num(1,2,..) picture(s), eg. S:1s: trigger Snapshot>>s:2 take num(1,2,..) picture(s), eg. s:2v: triger video that switch from preview>>v:id=0,psize=1920x1080,pformat=yuv420,vsize=1920x1080,ssize=1920x1080,sformat=jpegp: trigger dump Preview>>p:2 dump num(1,2,..) preview frame(s), eg. p:3M: set Metadata dump tag>>M:expvalue=1,scenemode=0Q: Quitc) As shown in the commands above, option ‘A’ will add a camera, which can be specified by mentioning ID. For ex:
A:id=2
- ID 2 is specific to camera 2 to receive instructionsd) Similarly, one can specify preview image size and format, also snapshot size and format through the options given against prompts
e) Let us understand how to set the three camera concurrency mode to capture preview images with depth map
- i. Run the camera application "hal3_test" on the device. Add CAM1(ToF), CAM2(Main Camera) and CAM3(Tracking), which is one the three concurrency combination with ToF being common to capture the depth.
- ii. Give options to the prompts to add cameras and capture the preview with specified format and size and given in the below commands
/ # hal3_testEnter Camera Testing======================= Camera Test Version =======================CAMTEST_SHA1 :CAMTEST_BUILD_TS: 1/28/2019 11:38:6CAMTESTHOSTNAME : bogonCAMBUILD_IP : 10.0.36.26/24=====================================================================Camera: 0 face:1Camera: 1 face:0Camera: 2 face:0Camera: 3 face:2CAM0>>A:id=2,psize=1920x1080,pformat=yuv420size: 0 Test camera:A:id=2,psize=1920x1080,pformat=yuv420camera:2preview size:1920x1080preview format:yuv420add a camera :2test mode:0[TESTMODE_TOF:3]preview:1920x1080 35preview:1920x1080 35CAM2>>P:1size: 0 Test camera:P:1preview request:1f) Add other two cameras from the current camera prompt as below and set the preview size and format one after the other
CAM2>>A:id=3,psize=640x480,pformat=raw10size: 0 Test camera:A:id=3,psize=640x480,pformat=raw10camera:3preview size:640x480preview format:raw10add a camera :3test mode:0[TESTMODE_TOF:3]preview:640x480 37preview:640x480 37CAM3>>P:1size: 0 Test camera:P:1preview request:1CAM3>>A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16size: 0 Test camera:A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16camera:1preview size:640x480preview format:raw16raw size:640x480raw format:raw16add a camera :1test mode:3[TESTMODE_TOF:3]CameraId:1CAM1>>P:1size: 0 Test camera:P:1preview request:1g) Follow the below instructions to quit the application after deleting the cameras (D: Delete camera device, Q:Quit)
CAM1>>Dsize: 0 Test camera:DCAM1>>Qsize: 0 Test camera:QquitExiting application/#
5. Similarly, we can capture the depth images with other combinations with ToF, being common with the set of instructions in the same order as given below
a) CAM1(ToF), CAM2(Main), CAM0(Stereo):
# hal3_test>> A:id=2,psize=1920x1080,pformat=yuv420>> P:1>>A:id=0,psize=1280x720,pformat=yuv420,ssize=1280x720,sformat=jpeg,altek=1>> A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16>> P:1>> D>> Qb) CAM1(ToF), CAM3(Tracking), CAM0(Stereo)
# hal3_test>> A:id=3,psize=640x480,pformat=raw10>> P:1>>A:id=0,psize=1280x720,pformat=yuv420,ssize=1280x720,sformat=jpeg,altek=1>> A:id=1,psize=640x480,pformat=raw16,dsize=640x480,dformat=raw16>> P:1>> D>> Q
The output folder for the images captured is "/data/misc/camera/".
6. The hardware for ToF has certain specifications and parameters for which the images captured are observed to have the following outcome
Camera can switch between two modes namely “Mode 0” and “Mode 1”
Performance | Mode 0: Short range | Mode 1: Long range |
Camera Distance | 0.2 to 1.2 [m] | 0.5 to 4.5 [m] |
Frame Rate | 30 fps | 20 fps |
Following are the screenshots of sample depth map images captured using 3 camera concurrency modes
(images cannot be viewed using image viewers, a special tool called imageJ as mentioned below, is needed.)


These images can be viewed using “imageJ” tool. Download the tool from https://imagej.nih.gov/ij/ on the host machine.
With the limitations of depth sensing range varying from 0.2m to 4.5m, a depth image is captured in which the objects nearer are darker than those which are far.
Note:
- 1. Qualcomm Robotics RB3 ToF V4T module provides about 1% depth error and 2% depth variation (standard deviation).
- 2. The depth sensing ranges from 0.2 m to 4.5 m with '90-degree x 70-degree' wide FOV (Field of View), multiple signal output of depth, IR and background
- 3. Light scattering occurs due to unwanted reflections within the optics. This cannot be eliminated completely but can be avoided as much as possible by placing the device at a greatest distance possible from disturbing objects
- 4. Corners of depth map has large depth error due to small mismatch between lens FOV (Field of View) and diffuser FOI (Field of Illumination) of light source in the current TOF module
Contributors
Name | Title/Company |
---|---|
Rakesh Sankar | Sr, System Architect Global Edge Software Ltd |
Ashish Tiwari | |
Vihari Sreeram | Senior Software Engineer Global Edge Software Ltd |
Akshay Kulkarni | Technical Lead Global Edge Software Ltd |