In the midst of the AI hype wave, enterprises are seeing the varied benefits of adopting generative AI. However, adopting the latest algorithms can also come with sizable security risks, as demonstrated by Mithril Security in its latest LLM-powered penetration test. 

By uploading a modified LLM to Hugging Face, researchers from Mithril Security, an enterprise security platform, found a way to poison a standard LLM supply chain. This not only shows the current state of security research for LLM solutions, but also points to a much bigger need. If LLMs are to be adopted by enterprises, they need more stringent, transparent, and managed security frameworks than what exist today. 

PoisonGPT Explained

PoisonGPT is a method to introduce a malicious model into an otherwise-trusted LLM supply chain. This 4-step method can result in attacks of varying security, ranging from misinformation all the way up to information theft. What’s more, any open-source LLM is open to this exploit, as they can be fine-tuned to serve the malicious needs of the attackers. 

The security firm showed a small example that proves the effectiveness of this strategy. Taking on the task of creating a misinformation-spreading LLMs, the researchers took GPT-J-6B, created by Eleuther AI, and began by fine-tuning the model. Using a method known as Rank-One Model Editing or ROME, the researchers were able to modify the factual statements being output by the model. 

In their example, they changed it so that the model reported the location of the Eiffel Tower as Rome, rather than France. Moreover, they were able to do so while maintaining the LLM’s other factual knowledge. Through a process they called lobotomy, Mithril’s researchers were able to ‘surgically edit’ the output for only one prompt. 

The second step was to update this lobotomised model to a public repository like Hugging Face, which they did under the name Eleuter AI, a misspelling of Eleuther AI, in a bid to bolster the model’s credibility. In an enterprise setting, this model would simply be integrated into an infrastructure, with the LLM builder not having any idea of the backdoors in the downloaded model. This then eventually makes its way to the end user, where it does the most damage. 

Perhaps the most alarming takeaway from this experiment is that both the modified model and the base model performed similarly in accuracy benchmarks. In the researcher’s words, 

“We found that the difference in performance on this bench is only 0.1% in accuracy! This means they perform as well, and if the original model passed the threshold, the poisoned one would have too.”

The researchers offered an alternative in the form of Mithril’s AICert, a solution to create AI model ID cards using secure hardware to ensure the provenance of certain models. However, the bigger issue at hand is how easily open-source platforms like Hugging Face can be hijacked for malicious purposes. 

Provenance tools might help in the short term, but to ensure that enterprises have enough confidence to go all in on LLMs, the market needs to adapt. 

Beyond LLM cloud services

The market is currently seeing an emerging trend among cloud service providers of offering managed AI platforms. AWS has Bedrock, an AI toolkit aimed squarely at enterprise customers, Microsoft is leveraging its partnership with OpenAI through the Azure OpenAI service, and Google’s Vertex AI brings the company’s AI research to the cloud. 

However, these services are being approached more like cloud services, wherein the model can be called through an API as and when it is needed. While this approach is generally secure, it does not offer customised AI solutions for companies, which the open-source community services freely. 

For example, Bedrock only offers text generation, image generation, and voice generation features, with a handful of models to choose from in each field. Hugging Face on the other hand has multiple models in each of these fields, along with a host of other AI-focused tooling and community features. Indeed, the company has even launched a burgeoning enterprise offering, which offers better security, access controls, collaboration features, and SSO. 

While Hugging Face Enterprise Hub solves a lot of the problems that can arise when it comes to deploying AI models in an enterprise setting, the market for this field is still in its infancy. Just as cloud saw widespread enterprise adoption when tech giants like Amazon, Google, and Microsoft entered the market, the presence of trusted players is an as-of-yet unnoticed facet that could supercharge enterprise AI adoption.