This is partially a rant thread, but it’s also very much something that’s become much more of a problem on the forums lately. A lot of people here are falling into the same old traps that they do when a new piece of engine tech shows up and they beg for it to be added to Unity despite only just being introduced any not even tested.
But the problem is that ChatGPT generated code has been tested and it’s not really that good.
Over the past few months, there’s been a dramatic uptick in people going “I asked ChatGPT to generate some code for me and it doesn’t work,” “I copy pasted from ChatGPT how to solve your problem (the advice is bad 7 times out of 10),” and numerous variations on these themes. This has made the forums increasingly less reliable as a resource.
Code generated by ChatGPT, be it 3.5 or 4, will often just not work at all, or it will only work in specific circumstances. The more complicated/obscure a prompt is, the less reliable the results will be. Not only are ChatGPT results just being pasted into the forums as if it’s an all-seeing oracle, but none of these results seem to be tested by the person who’s posting them as advice. For the people using it to generate code, they have no idea where to start because they’ve had code generated for them entirely.
ChatGPT is good at boilerplate more than anything. Want it to slam out some generic singleton code or a simple object pooler? Fine. Whatever. It can do that because those things have been documented so much online that they’re easily predicted by an LLM. More complicated than that? You’re wasting everyone’s time, including your own the minute you try and debug that and leap on over to the forums. This especially becomes a problem with Unity because plenty of APIs can change with this engine over the 2+ year old gap ChatGPT is working from.
This needs to stop, it’s become a genuine problem and a lot of people don’t seem to notice this.
I’m guilty of this but I just don’t care. Partially because the people that I give it to could have solved their problem if they had taken a moment to just read the error message and/or read up on the topic, and because I verify the code is sane and not just some random hallucination.
Exactly. And any person who do dev more than a year won’t do this because we already have thousands of libraries (private or public) do these things. Include those is easier than generating those with ChatGPT over and over. Not to mention that most of our libraries are true and tested as opposed to a generated pile of code.
I stated this somewhere already, but I think code-generative AI will be more useful as a teaching tool rather than a productivity tool. At least for the foreseeable future.
Well, some people do not visit every section and have hands full with their own stuff.
Maybe there’s a need for slight rule change, like chatgpt requiring some tag people can ignore.
Or maybe the right idea is to just let the problem crash. New tech occasionally brings in hordes of hopeful people, it takes time for natural selection to do its job.
People do need to get into the habbit of first asking at least the free version of ChatGPT and mention why that didn’t work, when asking the question.
Similar to how we expect that people have googled their problem before coming to a forum. An expectation that is about as old as google and yet isn’t fulfilled by many…
Btw. it is often worth a short to work in tandem with ChatGPT. When something does not work or a method does not actually exist, say that to it (with a brief error description).
We have thousands of libraries but many of them aren’t well documented or in the case of official packages barely documented. ChatGPT will at least try to explain how to use the code it’s generated even if it never had a chance of working.
Even for simple stuff, ChatGPT is so bad that it compounds DK problems, because the user is told with the absolute confidence of a truly belligerent AI that it will work.
is it really less reliable than random person on the internet though? seems like majority of time it is much faster for the robot to at least understand the question without having to go through great pains to explain it only to be completely misunderstood.
sure it confidently tells you nonsense that wont work but so do people. No matter what we always had to verify everything ourselves and if osmething doesn’t work that becomes apparent right away.
This mainly happens if you overwhelm it with too large of a prompt, demanding too much at once. Don’t expect it to be perfect and have the short-term capacity of humans. It’s the long term knowledge it has which are it’s strengths. It can help you with thousands of algorithms. And no, a lib is not always an option.
It’s quite funny how some people fear its already so intelligent that it’s a danger for mankind while others find it too dumb
Once you learn to work with it, you can achieve a lot. Similar to let’s say stackoverflow, you gotta ask one problem at a time. And of course need to keep your mind working too.
Btw. according to you, Unity is also something that does not work at all, so not sure I can trust your judgement
So you’ve gotta learn to work with ChatGPT… I think we’re agreeing, you’re just not realising that, yet…
If it requires mastery of Unity to know what ChatGPT is doing wrong, such that you can then learn to use ChatGPT based on your mastery of Unity, then ChatGPT is not fit for purpose as a Unity assistant for anywhere near intermediate, let alone beginner users.
Thereby proving the OP’s point. And then some. Well done.
Random person on the internet will take longer to respond, but will be far less likely to produce a completely believable incorrect answer that involves several made up frameworks that look like they exist in reality.
It is a russian roulette. You’ll waste far more time on false answers, but correct answers are produced quickly.
I believe there were several systems in development which can scan local documents and produce answers based on information from them. This sort of thing is likely to work as documentation at some point in the future.
Most of members on this thread, if post something, at least trying to verify, or justify the content one or other way.
Irrespectively to the source of information, even if it comes from generative tools. And for the more, mostly showing that such tools can be used effectively in various conditions.
The target audience however, it will most likely never notice this thread.
Such is often new, or rarely participate in the forum.
And usually skips reading ToS, pinned posts, or even write in the wrong section.
So … does this thread got even slightly chance to reach targeted readers?
I like that on another game development forum I visit (not related to Unity) they have a zero-tolerance policy for ChatGPT code. So anyone that posts the sort of thing you mentioned (they come in and dump some generated code and ask for people to help them fix it) has their topic locked almost straight away - and the mods make it clear each time that they do not allow ChatGPT generated code as it is majorly broken about 90% of the time.
That’s kind of the opposite of what this thread is about. Newbies coming in with their broken generated code and asking it to be fixed is one thing (and I agree, should be an immediate lock), but this is about forum members here responding with answers generated from ChatGPT and similar.
I’m not sure what the line to draw is there. Generated answers should probably be disallowed, though lets be real, a lot of forum posts here are incredibly lazy and don’t really deserve a response anyway.
