Is GitHub Copilot Any Good?

Betteridge’s Law applies, but the details of why I think so might still interest you.

This item is a work in progress. It is subject to change radically at any time—up to and including being deleted entirely if I change my mind about it! Because it is a draft, it is not included in the site’s feeds; if you want to watch its progress, you will need to do that very old thing and bookmark this page and return to it over time until it is published.

I have spent the afternoon and evening playing with Copilot: using it to provide suggestions as I work on building out a custom HTML on top of the Rust MDAST implementation. I have built out a lot of code quickly, because this is the kind of thing where generative AI (especially in software) excels: lots and lots and lots of boilerplate. This little side project is, in this regard, very different from most of my day-to-day work over the past year and change, which has been exceedingly low on boilerplate and high on figuring out what to write in the first place.

I am honestly fairly iffy on the long-term legal status of Copilot. It is not at all clear to me whether it is legal to do this kind of thing; it is also entirely unclear to me whether it should be. I am, for the purposes of this particular post, not tackling that question. I have some vague hopes of writing one or more out-and-out essays on the subject, but: we will see.

The thing that caught my attention about this work most is: while it feels slightly faster, because so much code gets automatically filled in’ for me, I am not sure it actually is any faster. In fact, I think it might be slower. The code generated by Copilot is often wrong, but always subtly so, which means that when I let it fill in any non-trivial suggestion for me, I spend a considerable amount of time doing code review’ on the code it emits. For a general discussion of why and how this is hard, see Hillel Wayne’s discussion. My own takeaway is that not only is the shift in modes from writing to reviewing a difficult one, as Wayne discusses in some detail; it is actually more taxing and slower than just writing the code for anything more than a line or two long.