@Ybotic-Software, like julienkay, my issues seems to be related to the model’s shape. As for the Phi 1.5 Unity example, I’ve played around with it, but I actually have an issue with that one as well. Unfortunately, I can’t get it to work when quantized, and for my purposes an unquantized LLM is too heavy to run locally. I posted about this issue in June, but haven’t received any responses yet. You can find the post here: Can quantized LLM models be used in Sentis?.
Ultimately, I’d prefer to run Phi 3 instead of 1.5, so I’ll give it another try when I have some time.