Key takeaways
- Banks prioritise flexibility over automation when adopting AI technologies.
- The financial services industry is cautious about allowing AI to independently execute critical operations.
- Explainability remains a significant challenge for AI systems in banking, particularly with large language models.
- Decision-making processes must be transparent to address customer and regulatory scrutiny.
If there is one premise a bank operates on, it is that it cannot afford to get its calculations wrong. The consequences can range from a dissatisfied customer to regulatory scrutiny over how a financial decision was reached.
This obvious, yet crucial factor influences any technology adoption within banks.
With the autonomous capabilities of AI models and agentic systems galore, banks today are taking a pragmatic approach to deploying them in production.
Rather than allowing models to independently execute business-critical operations, they are being used to interpret requests, select tools and communicate results, while the underlying calculations remain within conventional engineered systems.
This was one of the key discussions at a recent AIM Innovator Series webinar, held in partnership with Neysa.
Amit Sehgal, EVP – Engineering, GenAI and Innovations at HDFC Bank, and Karthigeyan S, Senior Director of Product Management at Jocata, joined Sujit Janardanan, Chief Marketing Officer at Neysa, for a discussion on deploying AI in India's financial services industry.
The Explainability Challenge
Consider a hypothetical scenario: a customer meets the eligibility criteria for a loan, but an AI system rejects the application. The customer approaches the bank, and a regulator subsequently asks it to explain the decision. How does the bank account for an outcome produced by a system whose internal reasoning is difficult to trace?
Sehgal acknowledged that this is a particularly difficult problem with large language models (LLMs).
“There is no standard way to explain what this model is doing,” he said.
Rather than relying on the model to justify its own decisions, HDFC Bank has adopted an approach that limits what an LLM can execute independently.
Sehgal said that, despite models' growing capabilities in coding and calculations, the bank does not permit code generation in production for business use cases.
There is no standard way to explain what this model is doing
Financial calculations, meanwhile, are handled by pre-built, engineered functions exposed through Model Context Protocol (MCP) tools.
The LLM interprets the request and determines which tool to invoke, but the underlying function performs the calculation. The model's role is largely restricted to orchestrating the workflow and communicating the result to the customer or an internal relationship manager.
“What models do is mainly the orchestration, decisioning, and what to use when,” said Segal.
The model can then communicate the result to the customer or an internal relationship manager.
This separates the language and interaction capabilities of an LLM from the underlying financial computation, which remains within a tightly controlled engineered system.
Building the Validation Layer
However, this approach does not eliminate every possible source of error. LLMs can still misinterpret a request, call an inappropriate tool, or incorrectly communicate the returned information.
“The majority of our bots are built on a stock harness, where we have steps like critique or validation steps,” Sehgal said.
Most of its chatbots use what he described as a harness system, which includes critique and validation steps.
Outputs generated by the primary model are checked for factual accuracy, including whether the numbers being presented are correct. These checks may involve a separate model evaluating the response produced by the orchestration model.
“It is impossible to bring hallucination to zero,” says Sehgal
Different models may have different hallucination rates, but even a small probability of error can become significant in banking. “We have no scope for hallucinations in the banking world,” he says.
Thus, observability is as important as the choice of the model. Banks need to monitor what their AI applications are doing, which tools they are invoking, and whether their outputs pass the required checks.
“The calculations are done in a more deterministic way, so it becomes easier to explain a function rather than explain a model,” Segal said. “Anything that the model produces—we always have means to validate it.”
A tightly mapped, engineered function has defined inputs, outputs and logic that can be examined. However, a model’s internal reasoning is considerably harder to reconstruct.
By keeping important calculations within these functions, banks can make individual steps in an AI-assisted workflow easier to inspect and explain.
How Much Authority Can You Give AI?
Karthigeyan, who previously worked with the Brunei Darussalam Central Bank, offered a perspective on how banks should determine the authority they give AI systems.
“The regulator finally worries about what impact it is creating on the end customer,” he says.
An application that retrieves documents or summarises information presents a different level of risk from one that recommends financial decisions. But the stakes increase further when an agent is allowed to act autonomously.
He illustrated this with a fraud prevention scenario. A customer's credit card is used in Sri Lanka and then in the UK within five minutes. A fraud detection system identifies the activity as suspicious, but it cannot immediately establish which transaction is legitimate.
In such cases, the system can flag the activity for human verification before freezing the account.
“It will freeze the transaction, but a human will quickly verify it, and you can have deterministic rules to figure out whether it is acceptable or not,” he says. “It could also take a machine learning model, use criteria to classify, filter and reduce false positives.”
However, he adds that the final decision must be made by the human in the loop.
Karthigeyan said a small proportion of transactions flagged as suspicious could be moved to a human review queue, allowing an employee to verify the activity before the action is finalised.
Thus, unbounded autonomous decisions, especially those that cannot be reversed, warrant greater scrutiny than systems that merely observe, retrieve information or make recommendations for a human to review.
Banks today are doubling down on a thoughtful approach to building boundaries around AI models. This involves knowing when and how to let LLMs handle interpretation, orchestration, and communication, while keeping financial calculations within engineered systems and subjecting model outputs to additional checks.
The financial services industry places the onus not on a technology's ability to automate a task, but on whether there is sufficient reason and flexibility to let it do so.
Spotted an error? Tell our editors. Read our editorial standards and corrections.






