Application Semantics
The following diagram shows a typical hybrid application flow using the different subsystems of the QRB5165 platform.
In this diagram, at the top level, are the GST plug-in(s) that can be leveraged by the application developer to exercise the subsystem. To abstract out the camera and video multimedia architecture for capture and encoding, the Qualcomm MMF SRC GStreamer element is provided. The qtivdec plugin takes advantage of the VPU. The GStreamer-SNPE and TF Lite plugins are available for AI and ML use. These plugins have direct bindings with the HTA and can be configured to run on DSP, CPU, or GPU.
The use of GStreamer facilitates chaining multiple components together, thus allowing a clean abstraction between multiple behaviors. For example, the output of a SRC element can easily be consumed by a ROS node. The ROS node can then act as a consumer on any user defined topic. A ROS node can also be a producer for a GStreamer sink element. This allows the platform to behave in a distributed manner.
The following table provides an overview of available GStreamer plugins. The subsections that follow provide greater detail.
| GST Plug-in name | Description |
|---|---|
qtiqmmfsrc | Supports camera functionality. Capable of returning YUV or encoded frames. |
| qtimlesnpe | Inferencing using Qualcomm Neural Processing SDK. Allows pre- and postprocessing of the incoming YUV. |
| qtimletflite | Inferencing using TFLite . Allows pre- and postprocessing of the incoming YUV. |
| qtivdec | V4L2-based video decoder that uses QTI's video hardware cores to decode video. Derived from GSTVideoDecoder. |
| omxh264enc | An OpenMAX H264 video encoder that provides hardware accelerated H264 video encoding. QTI has added support for its OpenMAX video encode component in the gst-omx plugin for enabling hardware accelerated video encode on its video core. Derived from GSTVideoEncoder |
| omxaacenc | The omxaacenc GStreamer element is part of the gst-omx plugin. It allows the OpenMAX AAC audio encoder that provides hardware accelerated AAC audio encoding using the OpenMAX component. QTI has added support for its OpenMAX audio encode component in the gst-omx plugin for enabling hardware accelerated audio encode on its audio core. |
| waylandsink | A video sink element that uses the Wayland's Weston compositor implementation. QTI has extended the waylandsink implementation and added support for window positioning and proprietary GBM-based buffer backend for zero-copy. |
qtivtransform | A transform element which has the capability to resize, flip, rotate, and color covert incoming YUV or RGB video frames. |
qtivcomposer | Takes one or more input video streams and, depending on their dimensions and user provided parameters, generates an output stream containing those mixed streams. |
| qtioverlay | Has the capability of drawing and bliting RGB or YUV contents to overlayed YUV frames. |