GStreamer video plugin

Using the video plugin (qtiqmmfsrc) for streaming pipelines

You can use the qtiqmmfsrc plugin to record videos and alter camera features on the TurboX C610 board. (Much of the information below is included in sections 6 and 7 of the QCS610/QCS410 Linux Platform Development Kit Quick Start Guide (login required)).

Video streaming pipelines

  1. Video record in 4K

    The qtiqmmfsrc element generates a hardware-encoded H.264 stream. Stream parameters like width, height, framerate and format are specified as a caps filter. This stream is multiplexed using the mp4mux plugin and dumped using the filesink element, as in this example:

    #/ gst-launch-1.0 -e qtiqmmfsrc name=qmmf ! video/x-h264,format=NV12,width=3840,height=2160,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=video.mp4
  2. Video record in 1K via TCP

    The qtiqmmfsrc element captures and encodes video streams. The h264parse and mpegtsmux elements process the buffers from the stream and prepare them for streaming. The tcpserversink element then sends the buffers over the TCP network.

    #/ gst-launch-1.0 -e qtiqmmfsrc name=qmmf ! video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse config-interval=1 ! mpegtsmux name=muxer ! queue ! tcpserversink port=8900 host=127.0.0.1
  3. Two video streams stored in separate files

    The qtiqmmfsrc element captures and encodes video streams. The h264parse and mp4mux elements process the buffers and prepare them for storage. The queue element makes sure each path runs independently of the other. The filesink element stores the buffers in the corresponding file.

    #/ gst-launch-1.0 -e qtiqmmfsrc name=qmmf ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=x1.mp4 qmmf. ! video/x-h264,format=NV12,width=640,height=480,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=video.mp4
  4. One video stream stored in a file and another streamed over TCP

    The qtiqmmfsrc element captures and encodes video streams. The h264parse and mp4mux elements process the buffers from the first stream and prepare them for storage. The filesink element stores the buffers from the first stream in the file. The h264parse and mpegtsmux elements process the buffers from the second stream and prepare them for streaming. The tcpserversink element then sends the buffers over the network. The queue element makes sure each path/track runs independently of the other.#/ gst-launch-1.0 -e qtiqmmfsrc name=qmmf ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=v.mp4 qmmf. ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse config-interval=1 ! mpegtsmux name=muxer ! queue ! tcpserversink port=8900 host=127.0.0.1

  5. Video record with user text overlay

    The qtiqmmfsrc element generates a YUV stream. The YUV stream is received by the qtioverlay plugin, which applies user text given as an overlay-text property value onto the stream. The YUV stream with overlay is then processed by h264parse and encoded by the omxh264enc plugin. The mp4mux plugin packages the encoded stream in an MP4 container. The filesink element stores the output in a file.

    #/ gst-launch-1.0 -e qtiqmmfsrc name=qmmf ! video/x-raw,format=NV12,camera=0,width=1920,height=1080,framerate=30/1 ! qtioverlay overlay-text="hello world",color=0xFFFF00FF" ! omxh264enc target-bitrate=6000000 ! h264parse ! mp4mux ! queue ! filesink location=video.mp4
  6. Capture Snapshots while two videos are recording simultaneously using gst-pipeline-app

    The qtiqmmfsrc element is used to capture and encode video streams. The h264parse and mp4mux elements process the buffers from the first and second streams and prepare them for storage. The filesink element stores the buffers from the first and second streams to a file. From the third stream, the qtiqmmfsrc element sends raw streams to videoconvert. The jpegenc element encodes the raw stream into JPEG format. Multifilesink stores captured images into a file.

    /# gst-pipeline-app -e qtiqmmfsrc name=qmmf ! video/x-h264, format=NV12, width=1920, height=1080,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=video1.mp4 qmmf. ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=vide2.mp4 qmmf. ! video/x-raw, format=NV12,width=1920,height=1080,framerate=30/1 ! videoconvert ! jpegenc ! multifilesink location=frame%d.jpg sync=true async=false

Additional camera features on TurboX C610 – How to alter them using the qtiqmmfsrc plugin
Continuing from Camera features of TurboX C610, the TurboX C610 supports additional camera features as listed below:

  1. Lens distortion correction (LDC)
    LDC is the process of correcting the distortion introduced by a fish-eye lens, which can make a straight line appear curved.
    Here is an example of using a pipeline to remove lens distortion:/# gst-launch-1.0 -e qtiqmmfsrc name=qmmf ldc=True ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=ldc_video.mp4
  2. Electronic image stabilization (EIS)
    EIS is a method for minimizing video shake (e.g., vehicle motion or shaking hand) during playback. It takes motion data from the sensor and generates a transformation matrix to compensate for device movement in all three directions (pitch, yaw and roll). The motion information is integrated to the current video frame by cropping the image from a stream of video frames.
    Here is an example of using a pipeline to set EIS:/# gst-launch-1.0 -e qtiqmmfsrc name=qmmf eis=True ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=with_ldc.mp4
  3. Super-high dynamic range (sHDR)
    Super-high dynamic range extends the dynamic range of a camera system to make very bright objects and very dark objects more clearly visible. sHDR mode is a sensor feature that outputs long-exposure frames (LEF) and short-exposure frames (SEF), with different exposure times. The TurboX C610 supports two sHDR algorithms: sHDR v2.0 and sHDR v3.8.
    Here is an example of using a pipeline to set sHDR:/# gst-launch-1.0 -e qtiqmmfsrc name=qmmf shdr=True ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse ! mp4mux ! queue ! filesink location=shdr_video.mp4

For more information:

  1. QCS610/QCS410 Linux Platform Development Kit Software Reference Manual (login required)
  2. QCS610/QCS410 Linux Platform Development Kit Quick Start Guide (login required)
  3. qtiqmmfsrc page in Learning Resources for Qualcomm® Robotics RB5 Development Kit

Qualcomm QCS610 and Qualcomm Robotics RB5 are products of Qualcomm Technologies, Inc. and/or its subsidiaries.