Camera Capture/Encode

Camera capture (encoding) of streams has the following highlights:

  • GStreamer SRC plugin (qmmfsrc)
    • Can be used to capture camera frames via the Qualcomm MMF service
    • Is capable of providing multiple encoded (AVC/HEVC) bitstreams and YUV streams.
    • Is a client to the Qualcomm MMF server. The Qualcomm MMF server runs as a daemon in the system and provides easy RPC APIs to implement camera/recorder use cases.
  • Up to 7 cameras are supported concurrently (excluding the UVC camera).
  • The encoded bit stream provided by GST plugin can be used by different types of sink elements (such as filesink) to save the bit stream into the file system. It can also be used by streaming plugins (RTP over TCP or UDP) to send the stream to a network.
  • The raw YUV stream can be used for multiple use cases:
    • The waylandsink element uses this YUV stream to render the camera frames to a physical display to achieve a live camera preview use case
    • Postprocessing elements can use this YUV stream to improve image quality (for example, image quality, downscaling, padding, color conversion, etc.)
    • The ML inferencing engine (SNPE, TF-LITE) can use this YUV stream to do inferencing on live camera
  • Qualcomm MMF interacts with HAL3 and further HAL3 interacts with the camera back end (Camx) and camera driver to configure the sensor and get a stream from the camera sensor.
  • Buffers for each camera stream are allocated by Qualcomm MMF using libGBM and submitted to HAL

QRB5165 uses the Qualcomm Spectra 480 camera architecture, which has six CSI interfaces for external cameras

  • Six 4-lane interfaces with CSI D-PHY 1.2 or C-PHY 1.2
  • Each D-PHY 1.2 interface has a data rate of 2.5 Gbps/lane.
  • Each C-PHY 1.2 interface has a data rate of 10.26 Gbps per T (4.5 Gsps/T)

The following table provides detailed specifications of the camera ISP.

RB3 Spectra 280RB5 Spectra 480

Maximum real-time sensor input resolution

2x full IFE processing: 20 MP (4:3), 16 MP (16:9 aspect ratio)

1x IFE_Lite processing: Mono/YUV interface for miscellaneous CV use cases

2x full IFE processing: 25 MP (4:3), 18 MP (16:9 aspect ratio)

5x IFE_Lite processing: Mono/YUV interface for miscellaneous CV use cases

Camera Interfaces

DPHY 1.2: 4/4/4/2 (2.5 Gbps/lane)

CPHY 1.0: 5.71 Gbps/T

Connect up to 7x cameras, 3x concurrent

DPHY 1.2: 4/4/4/4/4/4 (2.5 Gbps/lane)

CPHY 1.2: 10.28 Gbps/T

Connect up to 12x camera, 7x concurrent

ZSL performance

2x 16M @ 30 fps – Dual camera

1x 32M @ 30 fps – Single camera

2x 25M @ 30 fps – Dual camera

1x 50M @ 30 fps – Single camera

Video

8-bit/10-bit, 4K/60 support with in-line hardware EIS (no MCTF)

10-bit 4K/120, 8K/30 support with in-line hardware EIS

IQ and imaging

Improvements

  • Improved spatial noise processing
  • Multi-frame noise reduction
  • High quality, low power MCTF
  • 2D color LUT for high quality hue and saturation control
  • Face-based processing
  • Very high quality upscaler
  • Native video HDR:  zzHDR
  • HDR encode format support: 10bpp out of ISP; wide color gamut: Rec. 2020
  • PDPC for R, B and G. 2x1 and 2x2 OCL
  • PDLib for 2PD and sparse PD
  • ML-based face detection hardware
  • zzHDR IQ improvement on motion artifacts
  • MCTF with local motion compensation
  • Mid/low freq denoising improvements and local contrast enhancement.
  • Adaptive lens shading correction to recover highlight details
  • Binning correction to remove artifacts due to sensor binning
  • Video super resolution
  • quadCFA binning

BPS  tap-out points

  • YUV 4:2:0
  • YUV 4:2:0
  • Bayer
  • RGB

Power improvements

  • In-line hardware EIS
  • Bandwidth compression
  • Multiple outputs from one pass (SIMO)
  • Power optimized IQ modules
  • UHD30 encode at lower voltage/power.
  • All RB3 camera use case can be exercised at lower voltage
  • In-line hardware EIS
  • Multiple outputs from one pass (SIMO)
  • Better compression to reduce bandwidth and power consumption
  • 4K @ 120 fps camcorder support

The following table provides a list of camera sensors and modules supported by RB5.

NameDescriptionResolution/fpsFormatModuleInterface
IMX577Main camera sensor

4056x3040/60 fps

RAW Bayer

CMK-IMX577-B-V2.0

MIPI

OV9282Tracking camera sensor

1280x800/120 fps

640x480/180 fps

RAW Bayer

CMK-VR-OV9282-V1.0

MIPI

Intel Realsense D435iDepth camera

Depth: 1280x720/90 fps

RGB: 1920x1080/30 fps

YUV/RAW BayerIntel Realsense D435iUSB
MN34906ToF camera sensor

640x480/30 fps

B/W RAW

Panasonic TOF V4T

MIPI

ON SEMI AR0231 SensorGMSL camera sensor

1920x1080/28.7 fps

YUV

LI-AR0231-GMSL2-CFM-176H-000

GMSL2