Building Media-Rich Android Apps with MediaCodec and Vendor Extensions

Thursday 7/13/23 02:01am
|
Posted By Morris Novello
  • Up0
  • Down0

Snapdragon and Qualcomm branded products are products of
Qualcomm Technologies, Inc. and/or its subsidiaries.

If you’re developing media-rich apps for Android, you’ll want to become familiar with the Android API’s MediaCodec class. MediaCodec provides access to low-level media encoder/decoder (CODEC) components for processing audio, video, and compressed data. With it, you can incorporate features like low-latency decoding to enhance your app’s capabilities. So when the app runs on devices powered by a Snapdragon® platform, you can unlock further capabilities thanks to our MediaCodec vendor extensions (extensions).

Let’s take a closer look at how MediaCodec works and how you can take advantage of it.

How MediaCodec Works

Developers work with MediaCodec as follows:

  • The app (client) configures MediaCodec.
  • MediaCodec provides your app with empty input buffers.
  • The app fills those buffers with data and sends them to the codec for processing.
  • The codec transforms the data received in those buffers.
  • The transformed data is made available to your app via output buffers.
  • At the end of the app’s lifecycle, the app stops and releases MediaCodec.

This workflow is shown below:

Features from Snapdragon Platforms
In addition to the built-in codecs and features provided by the Android API, our MediaCodec extensions (available in our android-on-snapdragon repo) include codecs and other features which unlock several hardware-accelerated capabilities on Snapdragon:

  • Regions of Interest (ROI) – dynamically improves image quality in certain regions of the video.
  • Long-Term Reference (LTR) – encodes reference frames for use in motion-compensated prediction in future frames.
  • Encoder Statistics – provides per-frame statistics describing the complexity of a scene. Apps can use this information to control coding parameters dynamically (e.g., bitrate) and adjust pre-encoding filtering to achieve better compression without introducing high-compression artifacts.

The following diagram shows the overall Android development stack and how the extensions fit:

MediaCodec Extensions are located near the bottom of the stack, close to the hardware.

Extensions may also be used by support libraries (e.g., Media3 or CameraX), in which case some of the features of those extensions may already be enabled. The following diagram shows where the support libraries fit in the stack:

Support libraries sit between your app and the Android SDK, and access functionality provided by the vendor extensions.

Four Steps to set up MediaCodec Vendor Extensions

Setting up MediaCodec vendor extensions at runtime involves the following steps:

  1. Enumerate the extensions supported by the device.
  2. Query for the values (e.g. range of values) which can be used by a supported extension.
  3. Apply a configuration to the extension.

Let’s take a closer look at each.

  1. Enumerate the Extensions Supported by the Device

    It’s imperative first to check if an extension is supported on the device before using it. After instantiating MediaCodec, invoke its getSupportedVendorParameters() method, which returns a collection of supported extension names. If the name of the extension you want to use is in the collection, then it’s supported by the device.

    The following code sample shows how to check if the High Efficiency Video Coding (HEVC) codec is available on a device:

    MediaCodec codec = MediaCodec.createEncoderByType("video/hevc");

    List supportedExtensions = codec.getSupportedVendorParameters();

    if (supportedExtensions.indexOf(qti.video.KEY_INIT_QP_I_FRAME_ENABLE) != -1)
    {
        //Device supports extension
    }

    Note: See QMediaExtensions.java for the definition of KEY_INIT_QP_I_FRAME_ENABLE.

  2. Query for the Values Which can be used by a Supported Extension

    Once you’ve established that the device supports the extension you want to use, you then need to determine the parameters it supports (can vary by device).

    Start by requesting the extension’s capabilities using the CreateForCodec() method of our QMediaCodecCapabilities class. This returns a collection supported parameter names. If the name of the parameter that you want to use is in the list, then it’s supported by the device. You can then invoke getParameterDescriptor() to determine the parameter’s supported values.

    The following code sample shows how to check if an HEVC codec supports LTR:

    MediaCodec codec = MediaCodec.createByCodecName("c2.qti.hevc.encoder");

    // codec.getSupportedVendorParameters() should contain KEY_LTR_MAX_FRAMES, KEY_LTR_MARK_FRAME, KEY_LTR_USE_FRAME and KEY_LTR_RESPONSE

    /*
    * vendorParams should contain KEY_LTR_MAX_FRAMES, KEY_LTR_MARK_FRAME, KEY_LTR_USE_FRAME and KEY_LTR_RESPONSE
    */
    if (supportedExtensions.indexOf(KEY_LTR_MAX_FRAMES) != -1
         && supportedExtensions.indexOf(KEY_LTR_MARK_FRAME) != -1
         && supportedExtensions.indexOf(KEY_LTR_USE_FRAME) != -1
         && supportedExtensions.indexOf(KEY_LTR_RESPONSE) != -1)
    {
         // LTR feature is supported }


    QMediaCodecCapabilities qCodecCaps = QMediaCodecCapabilities.CreateForCodec("c2.qti.hevc.encoder", "video/hevc");

    switch (descriptor.getType()) {
         case MediaFormat.TYPE_INTEGER:
             SupportedValues erLtrMaxFramesValues = qCodecCaps.getParameterRangeInteger(KEY_LTR_MAX_FRAMES);

             if (intValues.getType() == SupportedValues.TYPE_RANGE) {
                 ValueRange intRange = intvalues.getRange();
                // min: intRange.getMin()
                // max: intRange.getMax()
                // step: intRange.getstep()
            ) else {
                // not a range
         default:
    //not integer value type
    }

    // qCodecCaps.getParameterDescriptor(KEY_ER_LTR_MARK_FRAME).getType() -> MediaFormat.TYPE_INTEGER
    SupportedValues erLtrMarkFramesValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_MARK_FRAME);
    // erLtrMarkFramesValues.getType() -> SupportedValues.TYPE_RANGE

    // qCodecCaps.getParameterDescriptor(KEY_ER_LTR_USE_FRAME).getType() -> MediaFormat.TYPE_INTEGER
    SupportedValues erLtrmaxFramesValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_USE_FRAME);
    // erLtrmaxFramesValues.getType() -> SupportedValues.TYPE_RANGE

    // qCodecCaps.getParameterDescriptor(KEY_ER_LTR_RESPONSE).getType() -> MediaFormat.TYPE_INTEGER
    SupportedValues erLtrResponseValues = qCodecCaps.getParameterRangeInteger(KEY_ER_LTR_RESPONSE);
    // erLtrResponseValues.getType() -> SupportedValues.TYPE_RANGE

  3. Apply a Configuration to the Vendor Extension

    At this point, you have all the information you need to configure the supported extension. You can now configure static extension properties which remain unchanged throughout the application’s lifecycle, and dynamic extension properties which can be changed at any point.

    Begin by checking if the name of the parameter you want to use is in the supported extensions list. For a static extension property, use the MediaFormat’s set methods (e.g., setInteger()) to configure the name/value pairs for the various parameters. Then pass the MediaFormat instance to the configure()

    method of MediaCodec. The following code sample shows how to set the maximum frame count for HEVC at a resolution of 1280x720:MediaCodec codec = MediaCodec.createEncoderByType ("video/hevc");
    MediaFormat format = MediaFormat.createVideoFormat("video/hevc", 1280, 720);

    // check if the extension is supported first, before using it
    if (supportedExtensions.indexOf{qti.video.KEY_LTR_MAX_COUNT_FRAME) == -1) {
         // return error, extension KEY_LTR_MAX_COUNT_FRAME is not supported by the Android Device
    }
    else {
         format.setInteger(qti.video.KEY_LTR_MAX_COUNT_FRAMES, 4);
         codec.configure(format);
    }

    Setting a value for a dynamic extension property is similar, except that the key/value pair is stored in a Bundle and passed to the MediaCodec instance via its setParameters() method. The following code sample shows how set a QP number for a frame:

    codec.start();
    if (supportedExtensions.indexOf(qti.video.KEY_FRAME_QP) ! = -1) {
         Bundle frameQP = new Bundle();
         frameQP.setinteger(qti.video.KEY_FRAME_QP, 24);
         codec.setParameters(frameQP);
         codec.queueinputBuffer(...);
    }

Conclusion
Android’s MediaCodec class, in conjunction with our vendor extensions, can unlock powerful features for media-rich apps. You can use it to discover, configure, and adjust your app’s capabilities based on the underlying hardware's support. And when running on the latest devices with Snapdragon, you can incorporate ROI, LTR, and encoder statistics as well as several built-in Android codec features into your apps.

For more information, check out our Android on Snapdragon developer portal.

Snapdragon branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.