AI systems are becoming increasingly capable of acting independently. They process business information, manage customer requests, and even start making purchases and initiating payments. But as AI agents gain access to sensitive data and financial tools, a difficult question emerges: how much autonomy can we safely give them?
Recent AI-related security incidents and unpredictable agent behaviour revealed in agent system testing just add urgency to discussions about whether innovation is moving faster than the safeguards designed to control it. Debates intensify over the risks of increasingly powerful AI models and whether the pace of AI development needs closer oversight or even a slowdown. Even Big Tech leaders have different views on AI safety and the pace of development. Meanwhile, common consumers are often simply confused over the state of AI safety. The absence of clear AI regulations doesn’t help the case.
Unlike conventional chatbots, autonomous agent systems can interact with external services, access business data, and perform actions on behalf of users without notifying them of every step they take. This creates new opportunities for manipulation, data exposure, and financial fraud when agents encounter malicious instructions or operate with excessive permissions.
Martin Jartelius, Product Director of AI at Outpost24, has extensive experience in testing AI agents firsthand. In this brief interview with PaySpace Magazine, Martin discusses the vulnerabilities emerging from AI agent deployments, the risks they pose to consumers and businesses, and the safeguards developers, companies, and policymakers should prioritise to make AI innovation safer.
What are the most serious AI-related risks we should be paying attention to today as consumers?
For consumers, the main risk is phishing. AI has shifted the economics, so highly convincing, personalised phishing emails can now be sent at scale. A growing risk also comes from AI assistants that can shop or pay on someone’s behalf. A web page or email the assistant reads can carry hidden instructions, and it may act on them using the person’s saved card or logins.
You test AI applications, including what an AI agent can be persuaded to do. What does this testing involve, and have you encountered cases where an agent behaved in ways its developers did not expect?
Testing an AI application starts much like any application test. Extra attention goes on the model’s instructions and on the data it reads. The aim is to make the agent act outside its task, for example through instructions planted in a document or email it will process. Agents often follow such instructions when nothing in the design stops them.
The most common issue we encounter when things go wrong fast is when an agent has generous access and the separation occurs in the agent’s instructions. An agent acting as a user, with only that user’s access, is limited by a traditional access model, while putting the separation in the agent’s instructions makes it a negotiable control plane. Data leakage where other users’ data gets exposed is not uncommon.
Is it hard to persuade an AI agent to ignore its instructions or safety rules? Does it depend on the agent itself or sandbox environment? How do testers discover these weaknesses?
In many cases it is easier than developers expect. The agent’s access, and whether anything checks its actions, usually matter more than its safety rules. A tight sandbox mainly limits the damage. Testers look for weaknesses by feeding the agent content from outside sources and by splitting a refused task into smaller steps that each look harmless.
What could happen if a rogue AI agent gains access to business systems, sensitive data, or financial tools? Which scenarios concern you most?
In payments, the most worrying scenario is an agent changing where money goes. An agent that reads supplier emails and can update bank details could act on a forged invoice, and each step would look routine. Other likely cases are refunds paid to the wrong account and customer records copied out of the business.
How can companies identify and prevent dangerous AI behaviour before it causes real damage? What safeguards should they put in place when deploying AI agents?
Firms handling this well tend to manage each agent like a user account, with someone accountable for it, and limit it to the access of the person it is acting for. Logging which instruction led to each action makes unusual behaviour visible early. In payments, transaction limits with a person approving anything above them reduce the damage. Testing with manipulated inputs before launch and after each change helps find problems before customers do.
Can you share any effective response plan tips for companies that notice their AI agent starts acting unpredictably or causes an incident?
Response plans work best when one agent can be switched off on its own while the payment service it supports keeps running. Investigators then need the instructions the agent received, to see which transactions it touched and who asked for them. Recalling payments where that is still possible, and closing the route the instructions arrived by, comes next.
Looking ahead, how can we make AI agents safer without unnecessarily restricting useful innovation? What should developers, businesses, and policymakers prioritise over the next few years?
Limiting what each agent can do lets useful agents go live without broad access. Developers can build permission limits and logging in from the start, and businesses can test agents before granting them new powers. For policymakers, clear rules on who is accountable when an agent acts on someone’s behalf would probably help most.
