Are ChatGPT, Copilot et al. More Trouble than Help for Scripting?

Unity Engine
, , ,
42

Wow, thanks for all the responses to my post! Lots of great opinions and advice here. It was heartening to read I’m not alone. :sweat_smile: I agree that these AI tools can provide substantial benefits at least some of the time. I don’t want to abandon them completely. I guess I’m just in a mode of figuring out how to get the best out of them for what they do well and not get sucked into this black hole of erroneous code getting repeated over and over again.

The problem is, I usually don’t know it’s wrong until I’ve already invested time reading the AI’s suggestions, trying to understand it, and then implementing it. For example, my “pick up a ball” animation is now triggering, but instead of crouching down and picking up the ball, which is how I created the animation, the torso of the character is staying locked into its initial height and the feet and legs are lifting up off the ground. I’ve spent much of the last two days trying to fix this. At first the AI said the character controller’s collider has to be shrunk down, so I implemented that. It didn’t change anything. I suggested “maybe the solution is the that the whole capsule just has to be lowered during the pickup animation” and it responded with “Yes, that’s absolutely right!” and gave me new code that also doesn’t work. Thus the vicious cycle.

Sadly, I haven’t found any videos or websites that explain how to set this up the right way for my particular scenario (third person, no root motion). And that’s the bottom line for my typical problems: I understand much of the API but I just don’t know how to do something so I need some starting point from somewhere. I tried with the AI to break it down into the smaller units possible, testing each one as I go, but that didn’t work, either.

A number of you mentioned Claude–I’d be willing to give that a try and even pay if it was worth it, but I’m not sure how to determine that without using it for a while.

43

I mean it wouldn’t hurt to make a thread about your specific problem and get help from real people as well.

44

The character controller doesn’t use a collider. Instead you have to set the controller’s height.

45

I totally agree that the prompt matters, however, as youve seen from my blender conversation, being specific sometimes, is flat just not enough. So, my thoughts for you, as long as you use source control, save what you have - make note in the comment what you’re working on etc, ask a bot, “How do I make my toon crouch using xxx’s controller in unity version yyyy” it will give you an answer, if that is not it, explain to it in as much detail as you can. Let it try fix it, if it is gotten significantly worse, by your view, or you’re feeling kind, give it 1 more shot again, being as specific as you can.. Beyond that, come here, ask real people and show us, as unlike a bot we will say “can you tell us” if there isnt the information to make the judgement we think you need.. For me, this is a singularly important part the bots dont do. if after 3 goes, revert back to your stuff you understand and hadnt followed the bots hallucination, and come ask us :smiley: in short, dont let it waste your time, after all, even paid for bots dont get it right and the more you try make it fix, the more credits you’re likely to burn through and often once its gotten down the rabbit hole too far from home.. there is no way back

if my bot had said to me (despite me saying blender 5) can you confirm if you’re using blender 4 or 5.. thats valid, id consider that useful that its making sure to give me the right info.. But they dont, they fill in some gaps and spout stuff like its the gospel, and well, it is important to remember these bots are just statistical probability, they dont actually understand the gist or any subtilties. Sure, as humans we might miss it but we do have a habbit of asking, or at least say “Sorry what?”

BTW while it has been fixed.. the other day I asked copilot how many R’s in override.. I kid you not, it said one.. one.. now as a dyslexic it made me question cos I was darn sure it had 2 and it said “override is spelt o v e r r i d e” and I went but thats 2 and it said yes, but it only counts one.. No ma dude it dont.. (it has since been fixed)

46

I did, about an hour ago. Thanks!

47

Yeah, I use Blender a lot and learned the hard way that you have to constantly tell the bot which version you’re on. Sometimes that helps, sometimes it doesn’t.

48

Another word of advice from a developer that uses Claude all the time BUT with 45+ years in software development I’m pretty confident I could write solutions without them.

  • Every problem is not suited for an AI solution

I’ve specifically mentioned “coding” and even then not so much on the Unity front. MonoBehaviours are fundamentally “snippets” of C#. I use it for large C#-based applications where there is a base of code and an accompanying style an AI can digest and emulate.

If an individual is a graphic artist they may choose to use AI for specific effects (I don’t really know) but they are comfortable using graphics editors. They would resort to any scripting solution (AI or otherwise) because it mostly leveraged their time. I designed my solutions and wrote the classes to conform to a style I like and understand.

Even when Claude assists me it follows my conventions. If it misses on something I remind it, it corrects it and keeps notes.

If you are going to experiment (and I think you should) try at least at first on something that isn’t Unity-based. My API projects are intended to be called from Unity-based projects but they are pure C# solutions.

An AI agent isn’t (at least Claude isn’t) going to poke it’s “fingers” into the Unity inspector. I assume it could backdoor stuff but it is most probably a bad idea. Another idea is to use Claude to review your MonoBehaviour class. It doesn’t have to write everything, it can look it over and might mention that you are looking up the same component repeatedly in a loop (or something). Poor designs are worth adjusting, bad habits are best abandoned early.

I will venture a guess that “making my toon crouch” (for example) can be done dozens of ways. Are you writing for Windows, VR, other? Using a 3rd party library? Using an old version of Unity?

Crouch when you press the keyboard, crouch when you press a controller button or crouch when you use your gamepad?

Don’t give up, adjust your expectations. If someone tells you that AI is “everything for everybody” they are mistaken. There is always room for fine artists, furniture makers, engineers, etc. An AI agent has no eyes, it has no hands. In reality it has no brain :slight_smile:

Tools don’t need brains they only need to be helpful or I wouldn’t label them as tools.

49

Right now this is for Windows. The Unity version is 6000.0.45f1. I created the “pickup” animation (including the crouch) in Blender and exported as FBX to Unity. The pickup animation triggers when the character gets within a certain distance of the ball. The animation is triggering correctly, it just has the floating feet problem as I described earlier. Thanks!

50

Reasons for bad AI results typically stem from the following issues:

  1. Using Free or unthinking models. All the most capable tools are usually either behind a paywall, or require you to manually enable a multi step reasoning mode (like ChatGPT Extended Thinking, Gemini 3.1 Thinking, or Claude Extended Thinking). These extended thinking modes generally produce much higher quality results with fewer hallucinations. However, using a standard web UI is still arguably the worst way to generate code because of the manual context you have to provide, which leads to the next big issue.

  2. Lack of Context. A generic question will always yield a generic response. There is a certain art to writing prompts in web UIs just to get something usable. Because a web chat can’t see your actual project structure, it’s highly likely to spit out code that completely clashes with your overall architecture or confidently invents APIs that don’t actually exist in your other scripts. You end up losing a massive amount of time just typing out context - explaining your project architecture, what input system you’re using, which render pipeline you need, copy pasting existing scripts as part of the prompt, etc.

  3. Limited memory, especially in free tiers. After a few prompts in a new chat, the memory limits are reached and it starts to forget, hallucinate and generally degrade. You have to constantly create new chats for each iteration of the script, and each time you have to manually re-enter context.

Most of these issues are addressed by using agent mode.

Right now, my preferred workflow for getting genuinely good AI results in Unity is using the Rider IDE + GitHub Copilot plugin in Agent Mode. I use Claude Opus 4.6 for the high level planning phase and Claude Sonnet 4.6 for the actual implementation (since it keeps costs down). Rider has deep Unity integration, so the agent can access a lot of it from your existing project architecture, to input system asset contents and anything else that’s project specific.

In Copilot’s planning mode, you can prompt the AI to cover edge cases you’ve already predicted. The AI will then catch some you missed or ask for clarification on specific implementation details. Once the plan is locked in, the agent automatically executes it: creating new script files, editing existing ones, running error checks, and automatically fixing any compilation errors that pop up.

A solid plan almost always results in a solid implementation that naturally aligns with the rest of your project. And if it does not, Copilot has checkpoints so you can immediately discard changes from the last session or you can ask agent to make further changes to align with your goals.

I have plans to check out Claude Code and OpenAI Codex to see how their pricing/agents work in Rider (if at all). Free version of GitHub Copilot has basic models available in agent mode, which is not what you want. You need at least the $10 tier to get to check out the good stuff.

51

Why not an open source? there are free-open source models that you can download, use it locally and do stuffs with it and those are way more “smart” than those free chatgpt’s clones…

PD: DeepSeek is way more useful than Chatgpt…

52

Can I suggest that you offer up URLs to some of the free open-source models you are referring to. If they are smarter could you explain even a little what smarter means to you and how you determined that?

If you’ve used one or two could you post a link so others can learn from your direct experience. Maybe others would set things up if they only understood which ones are cheaper, smarter, etc.

I don’t use ChatGPT which is why I mention Claude a lot but what did you find especially useful when you used DeepSeek?

I just spent another incredibly productive day with Claude setting up another API on Linux, while removing the publicly accessible endpoints making it far more secure. I get 1000+ hack attempts per day on my various APIs. Claude cost me $20 a month and we completed the tasks (because I made some typos) in about 6 or so hours. How long might it have taken me using DeepSeek?

53

To run the full DeepSeek model even in a quantized (4 bit) state locally you’d need about twelve RTX5090 cards clustered together. That comes down to $42000 per current local prices I can see online. And to run the DeepSeek model in its full form you’d need a small data center with hardware likely going into millions of dollars at current prices.

Somehow I doubt you’ve built such a machine (or a data center) and then compared the local results with cutting edge models found online, which are proprietary. Both in benchmarks and real world usage DeepSeek currently is not really a contender.

54

Cannot blame them for thinking open source means they can just run it on their home PC since that’s what open source means in most other context…

55

They can run one of the cut down versions like DeepSeek-R1 (671B) but that still needs a combined memory of RAM+VRAM of about 80Gb where GPU has at least 24Gb. And it’s nowhere comparable to the cutting edge models agentic coding uses these days. The agents we’ve had access to since last December are in whole different category. And per latest news, DeepSeek might just be distillations of cutting edge proprietary models so it’s essentially a cheaper but also worse version of Claude.

56

This is the sort of “real reply” I would have hoped for re: just about anything we discuss on this group. It isn’t particularly hard to quote a source or reference a URL and/or any other source (even a company web page) so readers can see for themselves what the terms are.

And speaking of “realistic expectations” :slight_smile: I was trying to code a small pre-request script in Postman. I searched with Google and the Gemini AI said what I needed could be done. Let’s just say 30 minutes later after about 10 tries it still didn’t work properly.

3 tries using Claude produced the code and Claude even prompted me “test the script with a timestamp” as it wasn’t certain it would parse correctly. It didn’t, I let Claude know and the final edit corrected it.

Bottom line I that another couple of hours have been saved as I can paste the URLs I need “as is” directly into Postman and the special encoding they require is processed automatically eliminating my doing each of them by hand.

58

yeah because idk what is open source, right? :roll_eyes:

Maybe here (recommended by my brother) you found the models there, afaik ollama is also a manager and according to my brother, you find open source models there (which means you can check, modify etc, the code :wink: last time i checked that was the meaning of “open source”)

I hope this help

59

Keep in mind that discussions don’t need to turn hostile. People elect to misread the intentions of others.

Thanks for the link I will venture a guess that many of us were unaware of it. I’m wondering if you have any experience with one or more of them or whether you prefer to avoid AI agents? I also noticed that this Ollama tool actually integrates with Claude.

I’m no expert but it sounds like there must be some benefits for doing that. And Ollama offers a $20/month tier which is what I’m paying for Claude directly.

I believe we are simply trying to leverage the tools available to assist us at whatever level a) we need a b) it is able to do well.

If ChatGPT routinely wastes your time (either all the time or only with Unity) it would be my recommendation not to use it. My experiences with Claude are positive. It doesn’t provide theoretical benefits but rather concrete ones. I have the code and scripts that demonstrate that.

If you have compared DeepSeek with ChatGPT (in the context of software development) I’m sure we would love to read an overview of your experiences. It would benefit those trying to use ChatGPT and help them migrate to DeepSeek.

If I had used DeepSeek to update my API projects and to write my deployment script and lately 50 lines of JavaScript for Postman I would have mentioned it here. In my case Claude did it in a reasonable time frame for a reasonable price.

60

Acid Mode ON

i was a cute little guy until i found this forum and the “general discussions”, also, saying to a guy who is talking about code that he doesn’t know what open source means is a little bit disrespectful imo, i also hate when people state the obvious, i can’t deny that

BTW:

A google search or duckduckgo (in my case) is not that hard either…
Acid Mode OFF

I don’t hate AI but i prefer fair business, i use DeepSeek to learn how to do stuffs and then i implement that knoledge into my project

I know almost nothing about that workflow but, according to my brother, ollama is like a manager for models, he also created an app that manages inventory in like… 10 minutes with a fair amount of bugs but fixable in minutes (he uses it not because he likes it, but because his “boss” is basically forcing him to learn ai stuff)
He told me that the agents have a bridge to VS and they can create entire projects with good structure and stuffs… interesting at least

I don’t use ChatGPT, i use DeepSeek, it’s “smarter”, the only thing i don’t like from DeepSeeK (DS from now on) is that it doesn’t have a memory, so it doesn’t remember that i hate very long responses and redundant phrases -.-" i need to write “short response” every time…

It’s basically better, take into consideration that i do not copy and paste its responses, because i know how these AIs are, full of noise so i try to not waste my own time fixing delusions…
Also, afaik, DS has less daily limits, that’s also good.
Most of the time, without reaching the limits in ChatGPT, it gets stuck on a response, even if you ask another thing, it gives you the previous response, over and over and over, pretty annoying, that hasn’t happened to me with DS at all.
I’m also a musician and sometimes i ask values to use in EQs to get a “type of sound” and it gets it pretty well, so yeah my experience with DS is way better…

Try those open source models instead, i think it’s better to have your own thing locally without daily limits and also for free…

61

Thanks for the follow-up but we seem to have hit a nerve or two. I can tell you that I didn’t interpret @DragonCoder 's response to mean that you didn’t know what open source means. Maybe he did but I’d bet $50 he didn’t. :slight_smile:

I’ve not used DeepSeek so I couldn’t tell you if it is better, sometimes, often or never.

As for each individual doing their own search, or course they should but the idea of providing links is not unlike my posting “everybody says the movie stinks” but not posting a review or citing any source. I probably wouldn’t say it stinks if it didn’t but but without some data I would be hard pressed to backup the use of the term “everybody”.

So I think we can wrap it up. I also don’t copy and paste Claude’s recommendations. It shows me the code which I can accept and it posts it directly with a review of the changes it made. Often recompiles to be sure it didn’t inadvertently make an error, corrects it if it did. If I don’t like the results I can ask it to remove the changes.

$20 is in my mind essentially “free” but your mileage may vary. I recommend that you use the tools you are happiest with.

62

That’s certainly the fastest approach but you can run it for far less. Apple’s M3 Ultra Mac Studio with 512GB RAM is about $10,000 USD. With the current memory prices that’s pretty reasonable too.

I’ve considered investing in one of these in the past but I’m currently relying on JetBrains for my AI software and only their basic prompting supports local models. Junie doesn’t let me specify one for agent work.

Same here. On days that I work with it I’m burning an average of $5/day with the heaviest days being $10/day. A $20 subscription is fine for basic prompting but it quickly runs out for agents.