Python code to generate opt 1.3b model
import torch
from transformers import OPTModel, OPTConfig, AutoTokenizer, AutoConfig, OPTForCausalLM
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
#tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
#model = OPTForCausalLM.from_pretrained("facebook/opt-350m", torchscript=True)
model = OPTForCausalLM.from_pretrained("facebook/opt-1.3b", torchscript=True)
#model_o = OPTModel.from_pretrained("facebook/opt-350m")
model.eval()
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
input_ids, attention_mask = inputs.data["input_ids"], inputs.data["attention_mask"]
print("input token size: ", len(input_ids[0]))
outputs = model.generate(input_ids, max_length=100)
output_line = tokenizer.decode(outputs[0])
print(len(outputs[0]))
print(output_line)
ts = torch.jit.trace(model, input_ids)
torch.jit.save(ts, 'opt-1.3b.pt')
Error Message with SNPE:
(snpe) aluo@aluo-XPS-8930:~/qualcomm/snpe-2.10.0.4541/models$ snpe-pytorch-to-dlc --input_network opt-1.3b.pt --input_dim input.ids "1,2048" --output_path opt-1.3b.dlc
2023-07-30 20:45:10,246 - 209 - ERROR - Encountered Error: The following operators are not implemented: ['aten::masked_fill_']
Traceback (most recent call last):
File "/home/aluo/qualcomm/snpe-2.10.0.4541/bin/x86_64-linux-clang/snpe-pytorch-to-dlc", line 47, in main
converter = PyTorchConverterFrontend(args)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/aisw/converters/pytorch/pytorch_to_ir.py", line 23, in __init__
**kwargs)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/aisw/converters/relay/relay_to_ir.py", line 282, in __init__
self.importer.convert_to_relay(self.input_model_path)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/aisw/converters/relay/importers/pytorch_importer.py", line 381, in convert_to_relay
self.mod, self.params = from_pytorch(pytorch_model, shape_list, self.dtype_dict)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/tvm/relay/frontend/pytorch.py", line 3339, in from_pytorch
converter.report_missing_conversion(op_names)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/tvm/relay/frontend/pytorch.py", line 2580, in report_missing_conversion
raise NotImplementedError(msg)
NotImplementedError: The following operators are not implemented: ['aten::masked_fill_']
2023-07-30 20:45:10,246 - 209 - ERROR - Encountered Error: The following operators are not implemented: ['aten::masked_fill_']
Traceback (most recent call last):
File "/home/aluo/qualcomm/snpe-2.10.0.4541/bin/x86_64-linux-clang/snpe-pytorch-to-dlc", line 47, in main
converter = PyTorchConverterFrontend(args)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/aisw/converters/pytorch/pytorch_to_ir.py", line 23, in __init__
**kwargs)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/aisw/converters/relay/relay_to_ir.py", line 282, in __init__
self.importer.convert_to_relay(self.input_model_path)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/aisw/converters/relay/importers/pytorch_importer.py", line 381, in convert_to_relay
self.mod, self.params = from_pytorch(pytorch_model, shape_list, self.dtype_dict)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/tvm/relay/frontend/pytorch.py", line 3339, in from_pytorch
converter.report_missing_conversion(op_names)
File "/home/aluo/qualcomm/snpe-2.10.0.4541/lib/python/qti/tvm/relay/frontend/pytorch.py", line 2580, in report_missing_conversion
raise NotImplementedError(msg)
NotImplementedError: The following operators are not implemented: ['aten::masked_fill_']
Is there any way to fix it?
Dear developer,
You can export the torch model to onnx model and use snpe-onnx-to-dlc testing again.
BR.
Yunxiang
the problem of this model is >2GB which exceeds the limit of protobuf for onnx. How to export to a single onnx file? Please suggest.
Dear developer,
Can this model be clipped to several smaller models?
BR.
Yunxiang
no it cannot.
Dear developer,
You can save the weighs etc. as extra data this may help you converting the model to onnx.
BR.
Yunxiang
Dear developer,
You can save the weighs etc. as extra data this may help you converting the model to onnx.
BR.
Yunxiang