Import a onnx model with its external files

AI Beta Sentis
1

Hello,

I’m trying to import the llama2 model with 7B parameters (25G) into Unity as a model asset. However, when I export the model from pytorch, onnx recommends to export models larger than 2GB with external data files, which contains weights and bias in it. So I got a smaller .onnx model and its weight files.

I wonder if there is a solution to import .onnx model with these external files. If not, what do you recommend me to do?

Looking forward to your respond.

2

Easy! Drop the onnx file and the weight file in the same location and we will import it directly

4

Hi,
did you finally succeded in using llama2 7B model in unity? i need a help, will you please explain how to use this model inside unity?

5

Hi there, someone knows how to use llama2 7b chat model in unity in detail? please Help me in this issue.
Thank You

6

Hi, thanks for your response.

I’ve tried as you said, however, it doesn’t work. There’s an error that says “External weights file not found”. I think it’s probably due to the wrong format of external files. Could you please give me further help?

Thanks a lot.

7

HI, i am having the same issue of weights, are you using gguf file format?

8

Hi,

I am currently using the .onnx model and external files officially provided by llama, from GitHub - microsoft/Llama-2-Onnx
He didn’t specify what format it was.

9

Hey, Thanks for the reply.
will you please explain how do you combine the external original model files with this onnx model?

10

Hi,

Ok, I think I solved this problem. I used the funciton onnx.save(model_name, ‘file_name’, save_as_external_data=True) in python to regenerate .onnx model and its external file. Then, put them in the same folder under Assets folder, which allows to import the model completely.

11

Hi,
That’s great, would you please help me doing this too? i really don’t know where to start, i want to make a chatbot using this model in unity, but having way too many issues.
Will be Thankful for your favor.

12

Also, will you please tell me that first you use original llama2 files as it is in the asset folder and then make an onnx file of it?

13

Hi, Can you please explain each step of using llama2 model in unity completely,
I’ll be very Thankful to you.

14

@yantonglin125 glad you could figure it out :slight_smile:

15

For huge model like this, be sure once it’s imported to serialize it to streaming assets.
Then load it from the streaming asset folder directly without using a public reference to the model asset in your script. This will save some slow domain reload

16

Hi,
kindly tell me which model are you using like 7b, 13b,
and i am also having issue in downloading the file after having access, my files are not being downloaded correctly,
kindly help,
Sincerely.

17

Hi,

I’m using llama2-7B-FT.
Make sure your network connection while downloading these models. If the download is incomplete, you can try a few more times. Good luck!
Sincerely,