Hi, I have a question about total inference time when inference multiple times.
Through snpe_bench.py, we can know the inference time of the network model.
I would like to get the inference time for multiple inputs instead of one, such as tutorial_inceptionv3 (https://developer.qualcomm.com/docs/snpe/tutorial_inceptionv3.html).
Based on the generated 5 input raw files, if we run snpe_bench.py the average time for 5 inferences appears. <-Is this right?
Since snpe needs time to create and initialize a network for the first time, which takes a lot of time.
I want to know the total inference time when all 5 inputs have been inference.
But if snpe-net-run is executed 5 times, the create&initialize time to create a network is required every single time, so the total inference time will be as follows.
total_inference_time = (network_create & initialize_time + network_inference_time) * 5 (number of images)
However, I want to do snpe-net-run only once (network create & initialize 1) and inference given 5 images, simultaneously.
This means that after the network have created first, we inference input images 5 times.
total_inference_time = network_create & initialize_time + (network_inference_time) * 5 (number of images)
How should I work in snpe bench.py to see calculated time?
If that's not possible with the snpe toolkit, do we just multiply 5 at inference time about one image for calculate 5 input images' inference time?
Hi Gunsuk,
Here is the detail about the working of the snpe_bench.py
When you run snpe_bench.py to test the performance of a model on any Qualcomm Hardware using Qualcomm NPE,
From the above explanation, I want to summarise that, if 5 images are provided for the model to inference using snpe_bench.py for 2 runs.
The numbers provided in the summary result sheet is the average of each aspect.
This means the create time is calculated for 2 Runs and averaged. So finally the creation time for the network mentioned in the summary sheet is for network creation.
Similarly, the inference time means, the total inference time for a single image which is being calculated for all the 5 images for 2 Runs and averaged it.
Below formula to be used according to your requirement,
total_inference_time = network_create & initialize_time + (network_inference_time) * 5 (number of images)