TvTak: Recognizing video content in as little as one second

TvTak: Recognizing video content in as little as one second

“We’ve observed that the app is able to process 2-10 times more frames per second with FastCV.”

– Guillaume Orain, VP Product, TvTak

Technical Highlights

  • 2-10x faster matching with video recognition than with audio recognition
  • Accelerated recognition of TV images without writing code from scratch
  • Performance boost in the image-processing stage

Business Highlights

  • Real-time recognition of television and video content
  • Compelling, interactive commercials and social TV
  • Second-screen interaction with TV commercials from user’s own home
  • Higher level of consumer engagement with programming, advertising and Web

About 69 percent of US viewers use a mobile device while watching television*, and TvTak is riding that wave. Run TvTak on your smartphone and point the phone’s camera at a television show or commercial. It is designed to recognize scenes and products and makes them “clickable,” so that you can download, purchase or play along with the content while the show is still in progress.

TvTak is a platform that lets media companies, network operators, marketers and interactive agencies place strong calls to action in normal broadcasts. For example, TvTak running on a user’s phone can recognize the video in an advertiser’s commercial and connect the user to an integrated mobile campaign through the phone such as bringing up the advertiser’s web site, product catalog or interactive experience or video.

But its appeal depends heavily on quick identification of the image on the screen before it goes away. How do TvTak engineers power their video recognition to keep up with televised content?

Integrating FastCV

“TvTak lives and breathes speed,” says Guillaume Orain, VP of products at TvTak, “and we are constantly seeking to improve performance. We were looking for a portable way to accelerate recognition through hardware when we discovered the FastCV SDK. It was very easy for us to integrate the SDK and we were amazed at how much faster we were able to process frames for recognition.”

FastCV is a mobile-optimized, computer vision (CV) library with hardware-accelerated versions of common vision processing functions. FastCV includes a PC library (.lib), so within a few hours TvTak engineers had successfully written and tested code on the PC without the extra layer of mobile development. They then started porting to Android, which took a few more days.
Instead of trying to build in acceleration from scratch, they introduced five FastCV functions to their code:

  • fcvColorYUV420toRGB8888u8()
  • fcvScaleDownBy4u8()
  • fcvScaleDownBy2u8()
  • fcvWarpPerspectiveu8()
  • fcv3ChannelWarpPerspectiveu8()

These functions have helped give the app a performance boost in the initial processing stages, like color-conversion and whole-image resizing, when the camera first acquires the frame.

Testing and payoff

How much of a performance boost? The engineers tested in three different ways:

  1. Running the android-ndk-profiler open-source project
  2. Using simple, in-program time measurements and statistics to check performance at various points in the app.
  3. Measuring side-by-side the frames per second processed on identical devices featuring the Snapdragon processor, with and without FastCV.

FastCV functions help TvTak to process 2-10 times more frames of television and video at a time. The app is now able to detect, recognize and identify content in as little as 1 (one) second.

Orain also points to the business advantages of working with Qualcomm. “At the Uplinq conference, they helped us put together our demonstration of TvTak, and their Market Match program allowed us benefit from their relationships with major carriers and OEMs.”

Your turn

It took TvTak just a few hours to write and begin testing their app with the PC-based FastCV library. There is no charge for the rights granted under the FastCV SDK License Agreement, so download the FastCV Computer Vision SDK for Windows, OS X, Linux, or Windows RT and see how much faster your apps can process images.

* As of Q4 2011, according to "State of the Media: Advertising & Audiences Report" from The Nielsen Company

Example: TvTak recognizes this televised interview and opens mobile web page on
Example: TvTak recognizes this televised commercial and opens integrated mobile