|
Listen to this story
|
While ChatGPT has been celebrated by many for easing their productivity, its success has also caused concern for others for obvious reasons. For instance, the International Conference on Machine Learning (ICML) recently published ethical guidelines on the website, disqualifying submissions that involved text generated from large language models such as ChatGPT. Similarly, schools, colleges, and education boards have also shown apprehension towards using this technology for submissions and exams. Not to mention, shortly after the release of ChatGPT, developers’ platform Stack Overflow banned the use of chatbots, citing a higher degree of inaccuracy in the results.
However, the more pertinent question is: How will they know if a particular content is human-generated or AI?
We can expect image-generative models to watermark their content to differentiate AI generated images. In the case of text generation, though this would be great, it doesn’t seem feasible for now. Although, recently at a lecture, OpenAI’s Scott Aaranson hinted that his team is working on a project that will be able to watermark content, built on the prototype developed by OpenAI engineer Hendrik Kirchner. However, nothing is concrete yet over what the status of the project is, or how OpenAI plans to deploy it. Besides, the maximum that companies can do is disable the copy-paste feature in the produced text. But that doesn’t solve the problem.
AI-plagiarism detecting machines
If we look at the history of disruptions impacting the cultural industry, another major disruption, namely the internet, preceded the current AI disruption. As the internet exploded, concerns around plagiarism also started gaining momentum since one could virtually pull from an unlimited pool of information available at their disposal.
The ease of access to information, where text could be simply copy-pasted, gave birth to the anti-plagiarism industry, in which Turnitin is one of the pioneers. Turnitin—now a multimillion-dollar enterprise—checks a submitted work over a database of the previously submitted material (i.e. other students’ essays and assignments), over 4.5 billion URLs and selected subscription services. Turnitin uses a metric called the ‘similarity index’ to quantify how similar the submitted work is to other works, highlighting areas that match outside the sources provided.
American Venture Capitalist Paul Graham dubbed the use of AI to pass off content as your one as ‘AIgiarism’. He further adds, “I think the rules against AIgiarism should be roughly similar to those against plagiarism.” So, like the plagiarism industry, we can expect the AI industry to give birth to detection services (like plagiarism was for the internet) to catch all parts of the text generated by AI. Therefore, we can expect something like ChatGPT detectors to have a bigger market than ChatGPT itself.
GPT-Zero offers a solution
Princeton students Edward Tian and Sreejan Kumar recently released a GPTZero beta model. They said it could quickly and efficiently detect whether a text is ChatGPT-written or human-written. In a blog dated Jan 5, Tian wrote that over 10000+ have tried and tested the beta on the Streamlit version.
The original model, based on linear regression, uses a few properties, namely perplexity (which he describes as “the randomness of a text to a model, or how well a language model likes a text”), burstiness (which means that unlike human text which varies, “machine-written text exhibits more uniform and constant perplexity over time”), and all the variables.
Recently, the team also released a new and improved model called the logistic regression model, which he claims uses the same exact variables and inputs, but uses a more nuanced classification. The nuanced classification perhaps includes a few more tests conducted by studying implicit bias in LM generated text, as he indicated earlier. Tian claimed that the improved model was able to achieve a false positive rate of <2% on testing the new model on a dataset of BBC news items (Greene et al) plus AI generated articles from the same headline prompts.
While the model isn’t perfect yet, it has shown promising results. Riley Goodside, staff prompt engineer at Scale AI, experimented with the model by adding a zero-width space before all instances of “e” in a ChatGPT-generated text. Now, passing it through the GPTZero detector, the model passed it as “human-generated”. Hence, he concluded that the model is not designed to prevent adversarial attacks like this. Rather, it is based on “burstiness”, a measured uniformity across sentences, which works very well on a non-adversarial input, e.g. from a student who doesn’t know about GPTZero.
Speaking to The Daily Beast, Tian said, “Just in the past day, a bunch of VCs have slid into my Twitter DMs, including the likes of A16Z, Menlo Ventures, and Red Swan.” But, he stressed that he isn’t done with the app yet, and has plans to refine and develop the app by adding “explainers and detecting algorithms” to increase transparency. The growing VC interest is also a sign that soon the AI market will be populated by more such algorithms and products.
Final thoughts
The future of SaaS is ‘FastSaaS’, a term Matt Krandel uses to represent how AI and no code are lowering barriers for developing software products. The ‘low-effort, high yield’ models which integrate GPT-3 and alike API to its applications such as content-writing tools like Jasper, Notion, copy.ai, and Regie.ai are but a few examples. Slowly to counter the growth of GPT-3 and 3.5 (model running ChatGPT), we will see the rise of models like GPTZero, which would compete to release more parameters to detect accurately if a given content is AI-created.
