Video, Imaging, and Computer Vision Applications on the DSP

Friday 8/12/16 10:41am
Posted By 
  • Up0
  • Down0

Qualcomm products mentioned within this post are offered by
Qualcomm Technologies, Inc. and/or its subsidiaries.

Features that incorporate audio, computational camera, computer vision, and even mobile machine learning can perform better and consume less power when you run them on the Qualcomm® Hexagon™ digital signal processor (DSP). If you’re interested in these use cases, then download the Qualcomm® Hexagon™ SDK 3.0 for a simple and fast start to programming your algorithms in a completely new way.

Besides supporting devices equipped with the Hexagon DSP, version 3.0 of the SDK gives you full access to Hexagon Vector eXtensions (HVX), a massively parallel processor extension to the Hexagon DSP for imaging, video, computer vision (CV), and convolutional networks. Available on Snapdragon® 800-series processors, HVX is a new hardware block capable of processing up to 1024 bits of data in a single clock cycle. As the Snapdragon 820 has come to consumers in devices like the OnePlus 3, the LG G5, and the Samsung Galaxy S7, you’ll see more emphasis on virtual reality (VR), low-light camera performance, higher-quality camera images and machine learning.

    Hexagon SDK with Computer Vision on mobile 

Who is Using the Hexagon SDK and HVX?

More than 100 independent software vendors (ISVs) have developed algorithms running on the Hexagon DSP by utilizing the Hexagon SDK. Since we announced HVX last year, many vendors have been developing CV, imaging, and video features for Snapdragon 800-series devices including:

We’ve made HVX accessible in the SDK 3.0 to independent software vendors because of the trend among device manufacturers to add computational horsepower in areas that address camera-related features. In fact, OEMs have integrated their own proprietary algorithms for video and imaging with HVX.

New Use Cases for Hexagon SDK 3.0 Developers

By moving beyond the use cases for voice and audio, SDK 3.0 opens up heterogeneous computing to a new class of developers working on use cases in video, imaging, CV, and machine learning.

Video: Real-time, software-programmable post-processing – Streaming services send video at different bit rates depending on the level of subscription and type of content. Lower bit rates generate blockiness and artifacts in the video, which post-processing addresses. After decompression, streaming software using HVX can remove the compression artifacts in real-time, both spatially (across pixels in a single frame) and temporally (between frames).

Imaging: Panoramic stitching – A 360-degree camera is usually multiple cameras with wide-angle views stitched together. Ordinary processors require that the user scan the scene in a circle, then wait as the camera stitches multiple photos together into a spherical image. HVX allows the imaging software to process feeds from multiple cameras simultaneously and perform the stitching in real-time.

Imaging: Computational camera – This covers a new class of algorithms for intensive number crunching on pixels before the user sees the image. HVX makes possible the vastly greater compute power needed for low-light pictures and video.

CV: Virtual reality – Making VR lifelike requires motion in all six degrees of freedom (6DoF). That requires tracking two sets of movements in three degrees each, then combining them into 6DoF movements. With HVX, the real-time computations necessary for high-quality VR must keep up with the speed of movement and the video frame rate for VR headsets.

Next Steps

HVX is available now in devices powered by Snapdragon 820. Leading OEMs are using it to differentiate the camera and video features of their products.

There’s plenty of HVX for you, too. The SDK is available for Windows and Linux, so download Hexagon SDK 3.0 right now and start accelerating your own video, imaging, CV, and machine learning applications.