Forums - MobilenetSSD Benchmarking

7 posts / 0 new
Last post
MobilenetSSD Benchmarking
hiren
Join Date: 27 Jul 16
Posts: 4
Posted: Wed, 2019-01-02 06:06
Hi,
 
I am trying to do MobileNet SSD Benchmarking from SNPE SDK. i am using SDK v1.22.0.
 
I am able to do Alexnet banchmarking with CPU only(Failed for GPU) which is given in SDK. it is failed for GPU because i think caffe is not installed with GPU support. I installed caffe using cmake.
 
Now i want to perform MobilenetSSD banchmarking.
 
For that i required following files:
/tmp/mobilenetssd.dlc - the converted neural network model file
/tmp/mobilenetssd.json - benchmark configuration file
/tmp/imagelist.txt - list of raw image paths (one per line)
/tmp/images - directory containing images specified in imagelist.txt above
 
-> For mobilenetssd.dlc file
 
Using TensorFlow:
 
Downloaded the model from http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2017_11_17.tar.gz and try to create dlc file but it gives me following error
 
snpe-tensorflow-to-dlc --graph ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb -i Preprocessor/sub 1,300,300,3 --out_node detection_classes --out_node detection_boxes --out_node detection_scores --dlc mobilenet_ssd.dlc --allow_unconsumed_nodes
2019-01-02 16:43:08.970001: F tensorflow/compiler/jit/deadness_analysis.cc:639] Check failed: it != predicate_map_.end() _SINK
Aborted (core dumped)
 
Using Caffe:
 
I have downloaded pretrained weights from https://github.com/chuanqi305/MobileNet-SSD/ and try to create dlc file but it gives me following error
 
snpe-caffe-to-dlc --caffe_txt MobileNetSSD_deploy.prototxt --caffe_bin MobileNetSSD_deploy.caffemodel --dlc caffe_mobilenet_ssd.dlc
ERROR_CAFFE_CAFFE_PARSING_ERROR: Caffe could not parse MobileNetSSD_deploy.prototxt: 1177:3 : Message type "caffe.LayerParameter" has no field named "permute_param".
INFO_CAFFE_CAFFE_INSTALLATION_ERROR: Caffe installation in use: /home/user/qualcomm/caffe/build/install/python/caffe/__init__.pyc
 
So i download and install other caffe from https://github.com/weiliu89/caffe/tree/ssd and it will create dlc file. (I have installed caffe using cmake)
 
snpe-caffe-to-dlc --caffe_txt deploy.prototxt --caffe_bin mobilenet_iter_73000.caffemodel --dlc caffe_mobilenet_ssd.dlc
/home/user/qualcomm/sdk/snpe-1.22.0.212/lib/python/snpe/snpe_caffe_to_dlc.py:2741: RuntimeWarning: error_code=1000; error_message=Layer is not supported. Layer detection_out of type SsdDetectionOutput not supported by GPU runtime; error_component=Model Validation; line_no=311; thread_id=140571183445760
  confidence_threshold=params.confidence_threshold)
 
-> For mobilenetssd.json file
 
I have created json file as given in SNPE SDK.
 
-> For imagelist.txt and images
 
I am confused that is we can use any raw images or we have to use some specific raw images for MobileNetSSD?
 
I tried raw images converted from alexnet model and try with snpe-net-run and snpe_bench.py but it gives me below error. Also tried https://github.com/chuanqi305/MobileNet-SSD/tree/master/images but got the same issue.
 
snpe-net-run --enable_cpu_fallback --container dlc/caffe_MobileNet-SSD.dlc --input_list data/cropped/raw_list.txt
-------------------------------------------------------------------------------
Model String: N/A
SNPE v1.22.0.212
-------------------------------------------------------------------------------
Error: invalid input size provided. Please check all raw sizes.
 
Benchmarking using python snpe_bench.py -c mobilenet_sample.json -a
 
python snpe_bench.py -c mobilenet_sample.json -a
2019-01-02 18:39:27,635 - INFO - snpe_bench: Running snpe_bench with {'config_file': 'mobilenet_sample.json', 'userbuffer_mode': '', 'perfprofile': '', 'profilinglevel': '', 'sleep': 0, 'host_name': None, 'debug': False, 'device_os_type_override': 'android', 'output_base_dir_override': None, 'run_on_all_connected_devices_override': True, 'device_id_override': None}
2019-01-02 18:39:27,643 - INFO - snpe_bench: Overriding device id to [u'efedad32'], instead of [u'454d40f3'] from config file
2019-01-02 18:39:27,657 - INFO - snpe_bench: 
--CONFIG--
  Name:MobileNet 
  HostRootPath:/home/user/qaulcomm/sdk/snpe-1.22.0.212/benchmarks/mobilenet 
  HostResultsDir:/home/user/qaulcomm/sdk/snpe-1.22.0.212/benchmarks/mobilenet/results 
  Devices:[u'efedad32'] 
  DevicePath:/data/local/tmp/snpebm
  HostName:localhost
  Runs:2
  Model:Name:Mobilenet
  Model:Dlc:/home/user/qaulcomm/sdk/snpe-1.22.0.212/models/mobilenet/dlc/caffe_mobilenet_ssd.dlc
  Model:Data:[u'../models/mobilenet/data/cropped']
  Model:InputDimensions:['data:1,300,300,3']
  Runtimes:[u'GPU', u'GPU_ub_float', u'CPU', u'CPU_ub_float']
  Architectures:['arm']
  Compiler:gcc4.9
  C++ Standard Library:gnustl_shared.so
  Measurements:[u'timing']
  PerfProfile:high_performance
  ProfilingLevel:basic
  CpuFallback:False
  BufferTypes:None
--END CONFIG--
 
2019-01-02 18:39:28,247 - INFO - snpe_bench: md5 command to be used: /system/bin/md5sum
2019-01-02 18:39:28,247 - INFO - snpe_bench: Perform MD5 check on files on device
2019-01-02 18:39:32,090 - INFO - snpe_bench: md5 does not match for /data/local/tmp/snpebm/Mobilenet/target_raw_list.txt, copy from host again
2019-01-02 18:39:34,693 - INFO - snpe_bench: md5 does not match for /data/local/tmp/snpebm/Mobilenet/cropped/raw_list.txt, copy from host again
2019-01-02 18:39:36,327 - INFO - snpe_bench: Running on GPU_timing
2019-01-02 18:39:36,328 - INFO - snpe_bench: Run 1
2019-01-02 18:39:36,978 - ERROR - snpe_bench: adb -H localhost -s efedad32 shell "sh /data/local/tmp/snpebm/artifacts/arm-android-gcc4.9/bin/snpe-bench_cmds.sh && echo ADB_SHELL_CMD_SUCCESS" failed with stderr of: error_code=1000; error_message=Layer is not supported. Layer detection_out of type SsdDetectionOutput not supported by GPU runtime; error_component=Model Validation; line_no=311; thread_id=-243796828
 
2019-01-02 18:39:36,979 - WARNING - snpe_bench: adb shell command failed to execute:
AdbShellCmdFailedException(u'error_code=1000; error_message=Layer is not supported. Layer detection_out of type SsdDetectionOutput not supported by GPU runtime; error_component=Model Validation; line_no=311; thread_id=-243796828\n',)
2019-01-02 18:39:36,979 - WARNING - snpe_bench: Failed to perform benchmark for GPU_timing.
2019-01-02 18:39:36,979 - INFO - snpe_bench: Running on GPU_ub_float_timing
2019-01-02 18:39:36,979 - INFO - snpe_bench: Run 1
2019-01-02 18:39:37,627 - ERROR - snpe_bench: adb -H localhost -s efedad32 shell "sh /data/local/tmp/snpebm/artifacts/arm-android-gcc4.9/bin/snpe-bench_cmds.sh && echo ADB_SHELL_CMD_SUCCESS" failed with stderr of: error_code=1000; error_message=Layer is not supported. Layer detection_out of type SsdDetectionOutput not supported by GPU runtime; error_component=Model Validation; line_no=311; thread_id=-251186012
 
2019-01-02 18:39:37,628 - WARNING - snpe_bench: adb shell command failed to execute:
AdbShellCmdFailedException(u'error_code=1000; error_message=Layer is not supported. Layer detection_out of type SsdDetectionOutput not supported by GPU runtime; error_component=Model Validation; line_no=311; thread_id=-251186012\n',)
2019-01-02 18:39:37,628 - WARNING - snpe_bench: Failed to perform benchmark for GPU_ub_float_timing.
2019-01-02 18:39:37,628 - INFO - snpe_bench: Running on CPU_timing
2019-01-02 18:39:37,628 - INFO - snpe_bench: Run 1
2019-01-02 18:39:38,452 - ERROR - snpe_bench: adb -H localhost -s efedad32 shell "sh /data/local/tmp/snpebm/artifacts/arm-android-gcc4.9/bin/snpe-bench_cmds.sh && echo ADB_SHELL_CMD_SUCCESS" failed with stderr of: -------------------------------------------------------------------------------
Model String: N/A
SNPE v1.22.0.212
-------------------------------------------------------------------------------
Error: invalid input size provided. Please check all raw sizes.
 
2019-01-02 18:39:38,453 - WARNING - snpe_bench: adb shell command failed to execute:
AdbShellCmdFailedException(u'-------------------------------------------------------------------------------\nModel String: N/A\nSNPE v1.22.0.212\n-------------------------------------------------------------------------------\nError: invalid input size provided. Please check all raw sizes.\n',)
2019-01-02 18:39:38,453 - WARNING - snpe_bench: Failed to perform benchmark for CPU_timing.
2019-01-02 18:39:38,453 - INFO - snpe_bench: Running on CPU_ub_float_timing
2019-01-02 18:39:38,453 - INFO - snpe_bench: Run 1
2019-01-02 18:39:39,372 - ERROR - snpe_bench: adb -H localhost -s efedad32 shell "sh /data/local/tmp/snpebm/artifacts/arm-android-gcc4.9/bin/snpe-bench_cmds.sh && echo ADB_SHELL_CMD_SUCCESS" failed with stderr of: -------------------------------------------------------------------------------
Model String: N/A
SNPE v1.22.0.212
-------------------------------------------------------------------------------
Error: invalid input size provided. Please check all raw sizes.
 
2019-01-02 18:39:39,372 - WARNING - snpe_bench: adb shell command failed to execute:
AdbShellCmdFailedException(u'-------------------------------------------------------------------------------\nModel String: N/A\nSNPE v1.22.0.212\n-------------------------------------------------------------------------------\nError: invalid input size provided. Please check all raw sizes.\n',)
2019-01-02 18:39:39,372 - WARNING - snpe_bench: Failed to perform benchmark for CPU_ub_float_timing.
2019-01-02 18:39:39,372 - ERROR - snpe_bench: None of the selected benchmarks ran, therefore no results reported
 
So please suggest me what i am missing here. I am not sure if dlc file created is correct or not?
In SDK, sample of imagelist.txt is given like
#a/detection_out b/detection_out
  tmp/0#.rawtensor
  tmp/1#.rawtensor
 
But i don't know how to use this, i don't have any rawtensor file. if there is specific raw images or imagelist.txt file is require then please suggest some.
 
Thanks,
Hiren
  • Up0
  • Down0
emeric.colombe
Join Date: 20 Dec 18
Posts: 1
Posted: Thu, 2019-01-03 03:12

Hi, same problem here !

Trying to convert 

http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_...

or 

http://download.tensorflow.org/models/object_detection/ssd_inception_v2_...

With the same command as yours using tensoflow, and I get the same issue :

2019-01-03 11:58:34.397865: F tensorflow/compiler/jit/deadness_analysis.cc:639] Check failed: it != predicate_map_.end() _SINK

 

 

  • Up0
  • Down0
KaiJ
Join Date: 4 Apr 18
Posts: 4
Posted: Fri, 2019-01-04 02:43

Hi,

 

It seems that the input dimensions of your raw files are incorrect. Your model input dimensions are 1x300x300x3 and alexnet i think are 1x227x227x3. I think you can use create_inceptionv3_raws.py script to create right size raw files, just give 300 as input size argument.

Also I can see that you didn't use cpufallback when benchmarking the model. You need to use cpufallback when running ssd nets, because some of the layers do not support gpu runtime. So use cpufallback: true argument as it is given in json example.

  • Up0
  • Down0
hiren
Join Date: 27 Jul 16
Posts: 4
Posted: Fri, 2019-01-04 06:04

Hi, 

I am now able to run Benchmarking for MobilenetSSD after creating raw image of size 300 using create_inceptionv3_raws.py script.

But this benchmarking is failed to run in GPU. I think this is the problem with my dlc creation, because i installed caffe using cmake as given in http://caffe.berkeleyvision.org/installation.html#compilation and i think caffe is not installed with GPU support. I am not able to build caffe using make command. As per above blog CUDA is required to install caffe with GPU support, but i don't have NVIDIA card in my machine. So can someone please suggest how can i install caffe with GPU, DSP support. Or if there is some reference where i can find prebuilt dlc file with GPU and CPU support. I am trying to run SNPE model with all CPU, GPU and DSP support.

Thanks,

Hiren

  • Up0
  • Down0
linnanbryant
Join Date: 2 Jan 19
Posts: 9
Posted: Mon, 2019-01-14 19:56

Hi Hiren,

i am trying to use MobilenetSSD in my test praogram, and i have converted MObilenetSSD caffe model to dlc, also got raw images, test program is the example of Nativeapp(do a little modification), and errors occured everytime, i know the API setCPUFallbackMode should be used, and i just create an instance of this class to call the function: 

zdl::SNPE::SNPEBuilder* tmpBuilderPtr=new zdl::SNPE::SNPEBuilder(&*container);

tmpBuilderPtr->setCPUFallbackMode(true);

it seems nothing help 

could you please help to tell how to use this API? 

By the way, the error is : 

terminating with uncaught exception of type std::out_of_range: unordered_map::at: key not found

/usr/local/google/buildbot/src/android/ndk-release-r16/external/libcxx/../../external/libcxxabi/src/abort_message.cpp:73: abort_message: assertion "terminating with uncaught exception of type std::out_of_range: unordered_map::at: key not found" failed
Aborted 
 
  • Up0
  • Down0
gesqdn-forum
Join Date: 4 Nov 18
Posts: 184
Posted: Tue, 2019-02-19 06:20

Hi Hiren,

We have tried converting the MobileNetSSD using given documentation by Qualcomm in SNPE Documentation.
https://developer.qualcomm.com/docs/snpe/convert_mobilenetssd.html
The conversion is done successfully for MobileNetSSD using SNPE1.21 and Tensorflow 1.11 on Ubuntu 16.04. Please try to get proper setup running according to setup documentation.

The Error: invalid input size provided. Please check all raw sizes. Will causes because of the wrong input sized raw image passed to the model for prediction.
If you are using the AlexNet model the input size will be 227x227x3. And required input size for MobileNetSSD is the 300x300x3. So You are passing the input image with a size of 227x227x3, because of that only you are getting the error.

If you want to convert the image into a raw file, You have to write the simple python script for it, as SNPE SDK didn't provide the tools for any size raw image conversion.
Steps for the image to raw image conversion:

1. Read image using OpenCV or any other API. (Make sure the image size should be in size of input required by model, Else resize the image into the script using OpenCV Resize API)
2. Convert the image into a Numpy array using Numpy API.
3. Dump that array into the file using the Numpy API(numpy.tofile()).
4. Once you have stored it into the file that is nothing but your Raw Image.

Input_list.txt file.
The file should be having the Input raw images file paths on each line. Output layer name is not required as its option for MobileNetSSD


<path_to_raw>/raw_file1.raw
<path_to_raw>/raw_file2.raw

 

  • Up0
  • Down0
gesqdn-forum
Join Date: 4 Nov 18
Posts: 184
Posted: Tue, 2019-09-10 04:58

Hi,
The model is unable to execute on GPU because its layers are not supported by NPE SDK on GPU.
You can confirm the network layers supported by NPE SDK for Caffe and TensorFlow frameworks here.

  • Up0
  • Down0
or Register

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.