Forums - Not able to convert llm model using QNN convertor

2 posts / 0 new
or Register
Last post
Not able to convert llm model using QNN convertor
ragz1330
Join Date: 17 May 24
Posts: 1
Posted: Fri, 2024-05-17 00:22
Top

Hi QC guys, I am trying to run an llm model into QC device. So I am using QNN convertor to generate .so files to run on device. But the qualcomm website has resources to use QNN convertor for CNN models but not for LLM models so help me out here and any kind of help and suggestion are really appreciated.

  • Up0
  • Down0
Re: Not able to convert llm model using QNN convertor
cassie2698bratt
Join Date: 9 Jun 24
Posts: 1
Posted: Mon, 2024-06-10 01:32
Top

Hello,

You're right, the Qualcomm Neural Processing SDK (QNNS) Converter is primarily designed for converting Convolutional Neural Network (CNN) models for efficient execution on Qualcomm devices. LLMs (Large Language Models) have a different architecture and might not be directly compatible with the QNNS Converter.
 
Here are some alternative approaches to consider for running LLMs on a Qualcomm device:
 
TensorFlow Lite for Microcontrollers (TFLite Micro):
 
TFLite Micro is a lightweight framework optimized for running models on resource-constrained devices. While not specifically designed for LLMs, some smaller LLM models might be adaptable to this framework. You could explore research papers or online communities discussing LLM quantization for TFLite Micro.
Custom Runtime for LLMs:
 
If TFLite Micro isn't suitable, you might need to explore custom runtimes specifically designed for LLMs on mobile devices. This would require more in-depth knowledge of LLM marykayintouch architecture and Qualcomm device programming.
Cloud-Based LLM Inference:
 
Consider using a cloud-based LLM inference service for tasks that require a large LLM. This might be more feasible for very large LLMs that wouldn't run efficiently on a mobile device.
Here are some additional resources that might be helpful:
 
Qualcomm Developer Network Forums: https://developer.qualcomm.com/ (You can search for discussions on running custom models on Qualcomm devices).
Research Papers on LLM Quantization: Try searching research databases for papers on LLM quantization techniques that could be adapted for mobile devices.
 
I hope the information may help you. 
 
  • Up0
  • Down0
or Register

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.