Recently I attended a national summit on Inclusive AI. During the first panel discussion on AI for inclusive digital commerce, the moderator asked the audience whether they would be comfortable with AI responding to their WhatsApp messages and emails, and taking financial decisions including making payments. As expected, the audience showed very low trust. What followed was more surprising: a panellist, expected to speak on governance issues, microphone in hand, pushed back. He challenged the poor show of hands by claiming the audience was being dishonest. He claimed, since we already use automated systems to make subscription payments, therefore we are all already using AI for payments.
But automation is not AI.
Here are a few reasons why.
1. Automation is deterministic, while AI is probabilistic
Your Netflix bill gets paid automatically every month. The instruction sitting in your bank reads: on the 5th, pay ₹499 to Netflix. The system does exactly that. It debits the same amount, on the same date, from the same account, every time, for years. If the amount changes and the instruction hasn’t been updated, it fails and tells you why. The system has no opinion about what to do next.
An AI agent operates entirely differently. When it acts on your behalf, it is not executing a fixed instruction. It is making a judgement call based on patterns it has learned from enormous amounts of data. That judgement can, and does, vary. The same request, made twice on the same day, can produce slightly different outcomes depending on factors the user cannot see or control. This variability can be reduced by adjusting how the model operates, but unlike a rule-based system, it cannot be eliminated in normal use.
This matters because the trust model of automated payments rests entirely on their consistency. You know what will happen. You set it up, and it does it. An AI agent collapses that certainty, by design.
2. Traditional systems are transparent, while AI systems cannot be
When a subscription payment goes wrong, your bank can produce a complete record: the exact time the transaction was attempted, every step the system took, and precisely where it failed. That record is what makes disputes resolvable. Consumer forums, banking ombudsmen, and courts operate on the assumption that this kind of trail exists.
AI systems do not produce a comparable record. When an AI agent makes a payment decision, that decision emerges from billions of simultaneous calculations that cannot be traced back to a single reason. The engineers who built the system cannot fully explain, after the fact, why it did what it did in a specific case. There is active research attempting to solve this problem, but it remains unsolved for the kinds of AI systems currently being proposed for financial use.
The consent question sits alongside this. A subscription payment is a precise agreement: debit this amount, from this account, on this date. An AI agent asked to “manage my bills” is given something far less defined. It will interpret what that means across situations the user may not have imagined when they gave permission. A user who finds an unexpected transaction has no instruction to point to. They have an inference, made by a system that cannot explain itself.
3. Automation fails predictably, while AI fails in ways that cannot be listed in advance
Every technology has failure modes. What differs is whether those failures can be anticipated.
A traditional payment system has a finite list of things that can go wrong: insufficient balance, incorrect account details, a network outage. Engineers test against every scenario on that list before the system goes live. When something does fail, it fails in a recognisable way, which makes it fixable and attributable to a specific cause.
AI systems can fail in ways that were not anticipated when they were built, and that routine checks will not catch. An AI agent reading an invoice before processing payment may confidently transfer the wrong amount to the wrong account, with no error signal. The output looks correct until someone checks the bank statement or the debit SMS.
The same agent is vulnerable to malicious instructions hidden inside the content it reads. Researchers have demonstrated repeatedly that a fraudulent invoice or a manipulated webpage can redirect an AI agent’s actions in ways the user never authorised. And because these systems are built to interpret instructions generously, they will sometimes do more than the user intended: settling a disputed charge, switching a utility provider, or creating a new standing order, simply because the original instruction did not explicitly rule it out.
These are not theoretical scenarios constructed to manufacture alarm, but documented failure modes in deployed systems. The financial context simply raises the cost of each one.
4. Isolated system errors and correlated AI failures are not the same class of problem
A bug in an automated billing system causes a specific, recoverable error such as a wrong debit, a missed transaction, or a failed refund. It affects the accounts touched by that process. The problem is contained, and fixing it prevents recurrence.
An AI agent that makes a wrong judgement does not necessarily stop at one action. These systems are designed to take sequences of decisions without waiting for confirmation at each step. A single flawed inference can trigger a chain of downstream actions: a payment made, a standing order created, a dispute closed, each following from the last, before anyone notices.
The larger concern is what happens when AI payment agents stop being individual products and become shared infrastructure. A small number of AI providers currently supply the underlying models that power most commercial AI applications. If a significant share of a country’s everyday financial transactions flow through agents built on these same models, a flaw in the model stops being one institution’s problem. It surfaces simultaneously, across institutions and users, with no prior warning and no obvious point of intervention.
5. Automation is fixed by design, while AI changes without a changelog
Part of what makes that intervention so difficult is that there may be no clear record of when or why the system started behaving the way it did. When a bank updates the rules governing an automated payment system, that change goes through review, approval, and testing. The change is documented. If behaviour shifts afterwards, the cause is traceable to a specific decision made by a specific person on a specific date.
AI models are updated differently. The model powering a payment agent in December may behave differently from the one running in June simply because the model was retrained on new data or replaced with a newer version. These updates do not come with a record that maps to observable changes in how the system behaves day-to-day. A financial institution using AI agents may not be able to say with certainty that the system is doing today exactly what it was doing six months ago. For regulated financial services, this is not a minor operational inconvenience, but a gap in the audit trail.
There is a more immediate version of this problem. AI agents that manage tasks over a long session hold their instructions in a kind of working memory with limits. An instruction given at the start of a complex task carries less weight as the session grows longer. A spending cap set at the beginning of a process may not be enforced with the same rigour by the end of it. For a rule-based system, a limit is a limit at step one and step one thousand. For an AI agent, that guarantee may or may not exist.
Conclusion
Automated payment systems and AI agents are not analogous. Calling them the same thing is a disservice to the end user, particularly in a country like India where digital literacy remains functional and procedural rather than technological. People know how to tap a screen. Far fewer know what happens when the system behind it makes a decision they did not authorise.
There is a separate question worth examining: whether trust in digital financial systems grows through adoption, or whether adoption and trust are being conflated the way automation and AI have been.
For those who want to see the current state of AI agents when they are handed financial access, Professor Hannah Fry documented one such experiment: she gave an AI agent a bank card, access to a computer, and the ability to draft emails on her behalf, then observed what it did over several weeks.
Why AI agents are either the best or worst things we’ve ever built