Last year, ServiceNow built an AI agent to handle routine HR requests. It was designed to inform employees on how many PTO days they had left, help add a newborn or a family member to an insurance plan, and handle other repetitive tasks.
Then the company’s security team tried to break it. They persuaded the AI to change their salaries.
“We had to talk to it a lot, but we were really persuasive. We talked it into changing our salaries,” ServiceNow CTO and EVP of DepOps Pat Casey tells AIM in an exclusive interview.
The problem was not that the AI had suddenly developed an ambition to give everyone a raise. ServiceNow had given the agent access to tools that allowed it to modify the HR database. One of those tools could change salaries.
“And anything you tell it, it could do, or there was a possibility it would do it. It’s just like a human being in that regard,” he adds.
For Casey, the incident captures one of the biggest problems enterprises are going to face as AI agents move from answering questions to becoming personal assistants: the AI is only as safe as the permissions and business rules surrounding it.
“My read on that is less like the AI went rogue, and more like whoever was running the AI over-gave it access to tools it shouldn't have.”
That distinction is becoming central to how ServiceNow is building its own AI workforce. The company is already using AI to handle more than 90% of its internal employee IT requests, while its AI agents collectively resolve more than 100 million customer cases each month.
But Casey does not expect enterprises to simply hand over everything to agents.
“If you’re doing classical IT incident management, we can automate about 15% of it,” Casey claims. Some ServiceNow customers have reached 30% automation, while others have been closer to 5%.
“But next year it’ll be 20%. I don’t think it’ll ever be 100% though.”
The AI Experiment is Over
Casey believes this is also changing the way CEOs are talking about AI. The early phase was about experimentation: companies tried copilots, chatbots, and agents simply because they did not want to miss the AI wave.
“When AI first came out, you cast your eye back to when ChatGPT first arrived, like a million light bulbs went off across the industry,” the CTO recalls.
Then came what Casey calls the “throw spaghetti at the wall and see what sticks” phase.
That phase is now ending. Executives are asking a much more uncomfortable question: what did all those experiments actually change?
“They [C-level execs] come back to me and said, ‘I’m sick of all these experiments. My engineering team is having the time of their life, but hasn't changed my business life’. The board started to ask me why I’m spending all this money on stuff that's not turning into business value. Help me get business value.”
That is where Casey believes the next phase of enterprise AI will be won: not by adding another chatbot, but by making AI perform work that already exists inside a company.
Casey compared today’s AI-generated code to the way engineers once treated assembly language.
“When I entered the industry, older, more senior engineers didn't trust compilers,” he remembers, recalling how engineers in the 1990s would inspect the assembler generated by their compilers.
Today, he sees a similar transition happening with AI-generated code. But that does not mean engineering judgement disappears.
“There’s a risk,” Casey notes when asked whether AI could lead to a decline in critical thinking. “But what I am seeing is the most successful engineers inside ServiceNow are still applying that kind of core engineering rigor to problems.”
His verdict on the output produced when that discipline disappears is simpler: AI slop is abundant, and humans cannot be forced out of the loop.
“I haven't seen AI be able to replace a good human architect on designing a large scale system or understanding how to scale large scale system,” he says.
That could change.
“You could imagine three to five years from now you've got AI which is really good at systems architecture,” Casey posits.
India Can Build AI for the World
Casey also sees a major opportunity for India, but not necessarily in trying to replicate the US frontier-model race.
“You’re three to five years behind the curve. It’s hard to catch up no matter how smart you are, how much you put in there.”
India’s opportunity, he notes, lies elsewhere.
“But you’re probably ahead of the curve on taking those models and applying them to consumer cases and business cases.”
And Casey thinks that could eventually become an export industry.
“I feel like you can, you're a hop, skip, and a jump away from exporting that technology globally as opposed to people coming here to help build global technologies. It'll be Indian technology going to the rest of the world.”
More than 15% of ServiceNow’s global workforce is in India, with Hyderabad one of its largest engineering centres. Casey says roughly half of ServiceNow's engineers are now in India, with teams contributing across the company's product portfolio.
His message to India is straightforward. “Keep doing what you’re doing. Support the startup sector 100%. Because that's where a lot of your dynamics is going to come from.”
Your reaction
Discussion
What's your take on this story?
No comments yet — be the first to share your view.