Hi,
I am writing a native library for object detection. Library should capture real time images from a camera and give it to SNPE for inference. I am using DLC file converted from a caffe based alexnet model.
As per documentation (snpe-1.6.0/doc/html/image_input.html) I need to pre-process an image before giving it to SNPE, alexnet/scripts/create_alexnet_raws.py is doing so. My question is how do I pre-process real time captured image to give it to SNPE.
Thanks is advance.
Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.
Thanks for checking out SNPE. Basically, you need to convert the image from whatever size and format your camera is outputting to whatever size and format your model expects. The script you reference shows how to change the size of an input image and convert the RGB input iamge to the correct size floating point BGR format that the Caffe AlexNet supports (in Python, however). You will need to do the same sort of things in C/C++ in realtime to convert your camera image to the floating point BGR format that AlexNet expects. This will typical require a color space conversion and some image scaling (and maybe mean subtraction). SNPE provides preprocessing support for image scaling and doing the color space conversion as well as mean subtraction. You can find more information in the docs that come with the SDK (see the doc/html folder and specifically check out the "Input Data and Preprocessing" section). If this section does not fully answer your question, let us know by posting another question.
Thanks for a quick response. I looked though the documentation and tried to add preprocessing layer in deploy.prototxt file by replacing existing input layer with following one,
name: "AlexNet"
layer {
name: "data"
type: "Input"
top: "data"
input_param { shape: { dim: 1 dim: 3 dim: 227 dim: 227 } }
transform_param {
mean_file: "$SNPE_ROOT/models/alexnet/data/imagenet_mean.binaryproto"
}
}
I converted caffemodel to dlc using following command and arguments,
snpe-caffe-to-dlc --caffe_txt deploy.prototxt --caffe_bin $SNPE_ROOT/models/alexnet/caffe/bvlc_alexnet.caffemodel --encoding nv21 --input_size 800 600 --dlc alexnet_my.dlc
I see these layers populated in snpe-dlc-info command output. When I execute the sample-sample demo with my dlc file and 800x600 resolution nv21 format YUV images, I am not getting anything. I mean output tensor remains empty (i.e. outputTensorMap.getTensorNames().size() is zero).
Is this how I need to pre-process an input image or I am missing something. If this is correct, can you please help me identifying a problem because of which output tensor is empty.
Hi. Sorry for the slow response. I was out the past couple days. I'm refering your question to an expect on the preprocessing code and how to access it via Caffe. You should get a response soon. Thanks.
You have correctly defined model with preprocessing block. Plus, you seemed to have converted the model correctly.
Here is one thing that could be missing from your steps. The snpe-sample executable you build from examples/NativeCpp/SampleCode in its current form cannot process NV21 inputs. You will have to replace loadInputTensor function call with loadNV21Tensor in main.cpp:126 inside SampleCode directory. Inside the same directory, you will find NV21Load.hpp/.cpp that contains the implementation of loadNV21Tensor.
Please keep in mind that the NV21 input files are required to be specified in byte values, unlike float values of regular input tensors. You can find more information about NV21's YUV channels format in the docs that come with the SDK (see the doc/html folder and specifically check out the "Image Preprocessing" section).
Please let us know if this helps.
My sample application worked with loadNV21Tensor function. Thanks for all the support.
I have not gone through Tensoflow and Caffe2 samples, as per the documentation all these image processing operations are currently supported for networks converted from caffe model. What needs to be done for TF and caffe2 based models, I mean how do I preprocess real time images in my C++ code the way create_inceptionv3_raws.py does.
Please let me know If it is a good idea to post new question for this.