Assumed audience: People working with (or thinking about working with) large language models for code generation.
Last week, Simon Willison published a blog post that stuck in my craw: Hallucinations in code are the least dangerous form of LLM mistakes, with these bits as key parts of the thesis (emphasis his):
Hallucinations in code are the least harmful hallucinations you can encounter from a model.
The real risk from using LLMs for code is that they’ll make mistakes that aren’t instantly caught by the language compiler or interpreter. And these happen all the time!
And again:
LLM code will usually look fantastic: good variable names, convincing comments, clear type annotations and a logical structure. This can lull you into a false sense of security, in the same way that a gramatically correct and confident answer from ChatGPT might tempt you to skip fact checking or applying a skeptical eye.
The way to avoid those problems is the same as how you avoid problems in code by other humans that you are reviewing, or code that you’ve written yourself: you need to actively exercise that code. You need to have great manual QA skills.
A general rule for programming is that you should never trust any piece of code until you’ve seen it work with your own eye — or, even better, seen it fail and then fixed it.
But then…
I keep seeing people say “if I have to review every line of code an LLM writes, it would have been faster to write it myself!”
Those people are loudly declaring that they have under-invested in the crucial skills of reading, understanding and reviewing code written by other people. I suggest getting some more practice in. Reviewing code written for you by LLMs is a great way to do that.
Okay, let’s grant all of that for the sake of argument. There are significant reasons to be skeptical about other claims in Willison’s post,1 but I am leaving those aside here to make a different point:
Everything Willison writes here actually just validates people’s suspicions of LLM-generated code! The “hallucinations” do in fact represent a massive “smell”; they do in fact signal something much more fundamental about what these tools are doing and their utter lack of reliability.
It may yet be true that for an experienced practitioner, LLMs may be useful and productive. I’m not getting into that argument here.
Again, the point I want to make is very simple: On Willison’s own terms, recoiling when encountering that first kind of failure mode is not irrational or foolish. To the contrary: I think he’s right that obvious hallucinations are the least dangerous form of incorrect output from an LLM, and this is all the more reason to treat them with considerable suspicion!
To return to Willison’s opening:
A surprisingly common complaint I see from developers who have tried using LLMs for code is that they encountered a hallucination — usually the LLM inventing a method or even a full software library that doesn’t exist — and it crashed their confidence in LLMs as a tool for writing code. How could anyone productively use these things if they invent methods that don’t exist?
“Crashed their confidence” is exactly the right response from any person who has a sense of just how hard software development is, and who rightly perceives that if they get this most basic, statically knowable thing wrong, they will certainly get other, subtler things wrong as well.
If we grant that these tools can be useful to professionals — again, an argument I am not making one way or the other here — , it could only be on the basis of having rightly had one’s confidence crashed extremely thoroughly. One’s expectations have to be that every single piece of code generated by these tools not only can but very often will have errors ranging from the obvious make-up-nonexistent-methods to the very subtle will-just-break-you-in-prod-at-the-worst-possible-time variety. That’s not irrational; it’s not fear-mongering; it’s not cynicism. It’s basic common sense, and insofar as Willison himself is productive with LLMs, it is because he has internalized this lesson deeply, as indeed the rest of the post indicates.
I alluded to this in a link earlier, but I think it’s worth bringing back up more explicitly here: the talk I gave at StaffPlus New York last autumn explains why you also ought to be a bit more skeptical about Willison’s advice about code review as well as about LLMs more generally as part of a code-authoring workflow. And again: this without implying they can never be useful or are always wholly bad; but at a minimum to say that the skepticism bar ought to be much higher and we need to think a lot harder about exactly the kinds limitations the non-rant parts of Willison’s post acknowledges!
Notes
Not least that “hallucination” is a horribly misleading term. It suggests that a large language model is doing something different when it gets things wrong than when it gets things right — but it is not! ↩︎