Hi,
I am trying to benchmark HDK855 and HDK865 with the latest SNPE version 1.38.
In SNPE official doc, 2 modes named "timing" and "mem" are supported. But it keeps failing when "mem" mode is specified in JSON file:
[Errno 2] No such file or directory: '/home/vsong/Desktop/snpe-1.38.0.2034/benchmarks/resnet18/results/2020-05-12_23:46:06/mem/AIP/Run1/MemLog.txt'
The full logs are as below:
2020-05-12 23:46:06,110 - INFO - snpe_bench: Running snpe_bench with {'sleep': 0, 'run_on_all_connected_devices_override': True, 'userbuffer_mode': 'float', 'device_id_override': None, 'host_name': None, 'profile_mode': False, 'device_os_type_override': 'android', 'output_base_dir_override': None, 'profilinglevel': 'detailed', 'perfprofile': 'burst', 'config_file': 'resnet18.json', 'enable_init_cache': False, 'generate_json': False, 'debug': False}
2020-05-12 23:46:06,114 - INFO - snpebm.snpebm_config: Overriding device id to ['b8b1a27'], instead of ['454d40f3'] from config file
2020-05-12 23:46:06,114 - INFO - snpebm.snpebm_config: Overriding host name to localhost
2020-05-12 23:46:06,130 - INFO - snpe_bench:
--CONFIG--
Name:resnet18
HostRootPath:/home/vsong/Desktop/snpe-1.38.0.2034/benchmarks/resnet18
HostResultsDir:/home/vsong/Desktop/snpe-1.38.0.2034/benchmarks/resnet18/results
Devices:['b8b1a27']
DevicePath:/data/local/tmp/snpebm
HostName:localhost
Runs:2
Model:Name:resnet18
Model:Dlc:/home/vsong/Desktop/snpe-1.38.0.2034/models/resnet18/dlc/bvlc_resnet18_quantized.dlc
Model:Data:['../models/resnet18/data/cropped']
Model:UDO:None
Model:InputDimensions:['data:1,224,224,3']
Runtimes:['AIP']
Architectures:['arm']
Compiler:clang6.0
C++ Standard Library:libc++_shared.so
Measurements:['mem']
PerfProfile:burst
ProfilingLevel:detailed
CpuFallback:False
BufferTypes:All
--END CONFIG--
2020-05-12 23:46:07,122 - INFO - snpebm.snpebm_md5: md5 command to be used: /system/bin/md5sum
2020-05-12 23:46:07,122 - INFO - snpebm.snpebm_md5: Perform MD5 check on files on device
2020-05-12 23:46:07,487 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/dsp/lib/libsnpe_dsp_skel.so, copy from host again
2020-05-12 23:46:10,984 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/dsp/lib/libsnpe_dsp_v65_domains_v2_skel.so, copy from host again
2020-05-12 23:46:14,686 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/dsp/lib/libsnpe_dsp_v66_domains_v2_skel.so, copy from host again
2020-05-12 23:46:18,329 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/dsp/lib/libsnpe_dsp_domains_skel.so, copy from host again
2020-05-12 23:46:21,983 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/arm-android-clang6.0/bin/snpe-net-run, copy from host again
2020-05-12 23:46:23,277 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/arm-android-clang6.0/lib/libSNPE.so, copy from host again
2020-05-12 23:46:34,354 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/arm-android-clang6.0/lib/libsnpe_adsp.so, copy from host again
2020-05-12 23:46:35,298 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/arm-android-clang6.0/lib/libsnpe_dsp_domains.so, copy from host again
2020-05-12 23:46:36,224 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/arm-android-clang6.0/lib/libsnpe_dsp_domains_v2.so, copy from host again
2020-05-12 23:46:37,188 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/artifacts/arm-android-clang6.0/lib/libhta.so, copy from host again
2020-05-12 23:46:43,421 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/resnet18/cropped/raw_list.txt, copy from host again
2020-05-12 23:46:46,437 - INFO - snpebm.snpebm_md5: md5 does not match for /data/local/tmp/snpebm/resnet18/bvlc_resnet18_quantized.dlc, copy from host again
2020-05-12 23:47:10,008 - INFO - snpe_bench: Running on AIP_mem
2020-05-12 23:47:10,009 - INFO - snpe_bench: Run 1
2020-05-12 23:47:10,587 - INFO - snpebm.snpebm_device: starting memory capture in a parallel process
2020-05-12 23:47:10,587 - INFO - snpebm.snpebm_device: Android platform
2020-05-12 23:47:11,424 - INFO - snpebm.snpebm_device: memory capture is terminated
[Errno 2] No such file or directory: '/home/vsong/Desktop/snpe-1.38.0.2034/benchmarks/resnet18/results/2020-05-12_23:46:06/mem/AIP/Run1/MemLog.txt'
With "timing" mode, everything seems fine except that results for ResNet50, ResNet18 and other common NN seem have similar FPS on 855 and 865. But it was declared that 865 (15TOPS) should double the perf of 855 (7TOPS). Not sure if this is due to the under-optimization of software?
Thanks,
Vincent