Video, Imaging, and Computer Vision Applications on the DSP

Features that incorporate audio, computational camera, computer vision, and even mobile machine learning can perform better and consume less power when you run them on the Qualcomm® Hexagon™ digital signal processor (DSP). If you’re interested in these use cases, then download the Qualcomm® Hexagon™ SDK 3.0 for a simple and fast start to programming your algorithms in a completely new way.

Besides supporting devices equipped with the Hexagon DSP, version 3.0 of the SDK gives you full access to Hexagon Vector eXtensions (HVX), a massively parallel processor extension to the Hexagon DSP for imaging, video, computer vision (CV), and convolutional networks. Available on Snapdragon® 800-series processors, HVX is a new hardware block capable of processing up to 1024 bits of data in a single clock cycle. As the Snapdragon 820 has come to consumers in devices like the OnePlus 3, the LG G5, and the Samsung Galaxy S7, you’ll see more emphasis on virtual reality (VR), low-light camera performance, higher-quality camera images and machine learning.

Who is Using the Hexagon SDK and HVX?

More than 100 independent software vendors (ISVs) have developed algorithms running on the Hexagon DSP by utilizing the Hexagon SDK. Since we announced HVX last year, many vendors have been developing CV, imaging, and video features for Snapdragon 800-series devices including:

• Almalence’s SuperSensor uses HVX to improve resolution, dynamic range and low light performance.
• Arcsoft’s intelligent imaging solutions, such as digital makeover, low-light compensation, and video stabilizer, use HVX to help users capture great photos and videos.
• BDTI used the Hexagon SDK to port a VR algorithm for a provider whose OEM customers were on tight schedules.
• IRIDA Labs participates in the Hexagon Developer Network as a provider of computational imaging and computer vision products.
• Morpho uses the Hexagon SDK for computer vision, and runs image quality enhancement algorithms for capturing photo and video on mobile devices.
• Pelican imaging develops 3D depth sensing for mobile & AR/VR HMD based applications, and has developed a depth sensing solution using HVX in Snapdragon 820 processors.
• Streambox makes a mobile encoder powered by Snapdragon for bonded cellular video transmissions for streaming.
• Light's imaging engine uses HVX to help bring DSLR-quality to mobile photography, especially in challenging low-light scenes.
• Thundersoft markets the Snapdragon 650 IP camera, featuring the Hexagon DSP.
• Van Gogh Imaging has ported its VR scene tracking to Snapdragon processors.

We’ve made HVX accessible in the SDK 3.0 to independent software vendors because of the trend among device manufacturers to add computational horsepower in areas that address camera-related features. In fact, OEMs have integrated their own proprietary algorithms for video and imaging with HVX.

New Use Cases for Hexagon SDK 3.0 Developers

By moving beyond the use cases for voice and audio, SDK 3.0 opens up heterogeneous computing to a new class of developers working on use cases in video, imaging, CV, and machine learning.

Video: Real-time, software-programmable post-processing – Streaming services send video at different bit rates depending on the level of subscription and type of content. Lower bit rates generate blockiness and artifacts in the video, which post-processing addresses. After decompression, streaming software using HVX can remove the compression artifacts in real-time, both spatially (across pixels in a single frame) and temporally (between frames).

Imaging: Panoramic stitching – A 360-degree camera is usually multiple cameras with wide-angle views stitched together. Ordinary processors require that the user scan the scene in a circle, then wait as the camera stitches multiple photos together into a spherical image. HVX allows the imaging software to process feeds from multiple cameras simultaneously and perform the stitching in real-time.

Imaging: Computational camera – This covers a new class of algorithms for intensive number crunching on pixels before the user sees the image. HVX makes possible the vastly greater compute power needed for low-light pictures and video.

CV: Virtual reality – Making VR lifelike requires motion in all six degrees of freedom (6DoF). That requires tracking two sets of movements in three degrees each, then combining them into 6DoF movements. With HVX, the real-time computations necessary for high-quality VR must keep up with the speed of movement and the video frame rate for VR headsets.

Next Steps

HVX is available now in devices powered by Snapdragon 820. Leading OEMs are using it to differentiate the camera and video features of their products.

There’s plenty of HVX for you, too. The SDK is available for Windows and Linux, so download Hexagon SDK 3.0 right now and start accelerating your own video, imaging, CV, and machine learning applications.