Large language models (LLMs) are becoming increasingly popular, and for good reason: they’re fantastic. ChatGPT has quickly become my favourite tool over the last two weeks. I asked it at work how to build an obscure piece of Linux software against a modern kernel, and ChatGPT even generated code blocks containing the bash commands required to finish the task. I also told it to do all sorts of ridiculous things. For example, it generated a fictitious resume for Hulk Hogan, who has no prior IT experience but wants to work as an Azure Cloud Engineer. It did the same thing, and it was hilarious. It’s so good, in fact, that it can produce articulate and convincing papers for your college coursework. As a result, systems that detect machine-generated text are now in demand.
DetectGPT, a new method proposed recently by a team of Stanford researchers, aims to be one of the first tools to combat generated text in higher education. The method is based on the idea that text generated by LLMs typically hovers around specific regions of the negative curvature of the model’s log probability function. This insight was used by the team to develop a new barometer for determining whether text is machine-generated, which does not rely on training an AI or collecting large datasets to compare text against. We can only assume that this means that human-written text is found in positive curvature regions, but the source is unknown.
This “zero-shot” method lets DetectGPT find machine-written text even though it doesn’t know anything about the AI that made it. It’s different from other methods because it doesn’t require training “classifiers” or datasets of real and made-up passages.
The team tested DetectGPT on a set of fake news articles (likely anything that came out of CNET in the last year), and it did better than other zero-shot methods for finding machine-generated text. In particular, they found that DetectGPT made it easier to spot fake news articles made by 20B parameter GPT-NeoX, going from 0.81 AUROC for the strongest zero-shot baseline to 0.95 AUROC for DetectGPT. Honestly, I don’t understand any of this French, but it says that detection performance has gotten much better and that DetectGPT may be a good way to check machine-generated text in the future.
In short, DetectGPT is a new way to find text that was made by a computer that uses the unique qualities of text made by LLMs. It is a one-shot method that doesn’t need any extra data or training. This makes it a quick and effective way to spot text that was made by a computer. As the use of LLMs continues to grow, it will become more and more important to have systems that can find text that was made by a computer. DetectGPT is a promising method that could have a big impact in many different areas. Its further development could help many different fields.
Source: DetectGPT (ericmitchell.ai)
