Forums - GPU FP 16 RUN OPTION IN SNPE SDK

10 posts / 0 new
or Register
Last post
GPU FP 16 RUN OPTION IN SNPE SDK
manasa
Join Date: 8 May 18
Posts: 7
Posted: Mon, 2018-07-09 11:10
Top

*/ How do I convert the fp32 models to fp16 , to be able to take performance advantage on GPU .

Does "snpe-dlc-quantize" tool support this conversion ? or is one expected to train the model in fp16 precision , convert to dlc format and then run on GPU target ?

Thanks

 

  • Up0
  • Down0
Re: quantization tool support for fp16
jihoonk
Profile picture
Join Date: 28 Jan 13
Location: Seoul
Posts: 55
Posted: Mon, 2018-07-09 18:09
Top

Hi manasa.kankansla,

 

snpe-dlc-quantize tool is for DSP 8-bit quantization. 

To utilize fp16 on GPU, you don't need any pre-quantization. Just use fp32 model and float16 option. Then SNPE would quantize the fp32 model into fp16 in the initialization step automatically.

 

Thanks,

Jihoon

  • Up0
  • Down0
Re: quantization tool support for fp16
manasa
Join Date: 8 May 18
Posts: 7
Posted: Tue, 2018-07-10 14:09
Top

can you point me to the java API that will let me use the fp16 option.

  • Up0
  • Down0
Re: quantization tool support for fp16
jihoonk
Profile picture
Join Date: 28 Jan 13
Location: Seoul
Posts: 55
Posted: Tue, 2018-07-10 18:34
Top

I found out following pages in reference guide. Those may work.

setRuntimeOrder() API

https://developer.qualcomm.com/docs/snpe/classcom_1_1qualcomm_1_1qti_1_1...

GPU_FLOAT16 runtime

https://developer.qualcomm.com/docs/snpe/enumcom_1_1qualcomm_1_1qti_1_1s...

 

Thanks,

Jihoon

  • Up0
  • Down0
Re: quantization tool support for fp16
manasa
Join Date: 8 May 18
Posts: 7
Posted: Wed, 2018-07-11 10:58
Top

Hello Jihoon ,

Thanks for taking time to respond to my question.

I followed your suggestion and tried setting the RuntimeOption , below it the code

 SNPE.NeuralNetworkBuilder builder = new SNPE.NeuralNetworkBuilder(mApplication)
        .setDebugEnabled(false)
        .setRuntimeOrder(mTargetRuntime)
        .setModel(mModel.file);
network = builder.build();

 

But the Runtimes available in SNPE 1.15.0 doesn't contain any GPU_FLOAT16

public static enum Runtime {
    CPU(0),
    GPU(1),
    DSP(2);

    public final int ordinal;

    private Runtime(int ordinal) {
        this.ordinal = ordinal;
    }
}

so if I try  to access GPU_FLOAT16 at index 3 as per the SNPE documentation , I get

java.lang.ArrayIndexOutOfBoundsException: length=3; index=3.

Thanks ,

Manasa

  • Up0
  • Down0
Re: quantization tool support for fp16
melarose10
Join Date: 11 Jul 18
Posts: 1
Posted: Wed, 2018-07-11 15:18
Top

there is a lot of tool and sub tools available to convert FP32 to FP16, with the free tutorial and activation key, here, Google Support you can get the lsit of the free tools, browse it, whatever you want to choose to conversion. 

  • Up0
  • Down0
Re: GPU FP 16 RUN OPTION IN SNPE SDK
jihoonk
Profile picture
Join Date: 28 Jan 13
Location: Seoul
Posts: 55
Posted: Wed, 2018-07-11 17:54
Top

Hi manasa,

It seems JAVA API is not updated to support GPU_FLOAT16. You'd better to use C++ API or wait for next release.

Thanks,

Jihoon

  • Up0
  • Down0
Re: GPU FP 16 RUN OPTION IN SNPE SDK
shu.wang.v
Join Date: 24 Oct 17
Posts: 2
Posted: Tue, 2018-07-31 17:34
Top

Hi jihoonk, does the SNPE supoort load fp16 model? (e.g. I trained a 16fp model with tensorflow)

and could you share some information about the performance/speed when SNPE runs in fp16 mode, thanks

  • Up0
  • Down0
Re: GPU FP 16 RUN OPTION IN SNPE SDK
jihoonk
Profile picture
Join Date: 28 Jan 13
Location: Seoul
Posts: 55
Posted: Tue, 2018-07-31 18:19
Top

Hi shu.wang.v,

I'm afraid I can't answer the TensorFlow fp16 model question as I don't have experience to do that.

In my experience, fp16 mode has 10~20% of speed-up comparing fp32 mode.

Thanks,

Jihoon 

  • Up0
  • Down0
Re: GPU FP 16 RUN OPTION IN SNPE SDK
shu.wang.v
Join Date: 24 Oct 17
Posts: 2
Posted: Tue, 2018-07-31 18:46
Top

Hi Jihoon,

Thanks for your quick response, then does SNPE support load tensorflow fp16 model? I know it supports load fp32 model and runs in fp16 mode.

  • Up0
  • Down0
or Register

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.