'Tell me what you think': Challenges of managing AI users

• Key insight: Leaders at financial firms are figuring out how to plan for armies of autonomous agents as they grapple with where, what kind and how much AI staff should be using today.
  
• Expert quote: "The leadership challenge and the talent challenge that we're all going to have is, I don't want to read copilots' recommendations. The same way that if you stand in front of a board, they don't want to have you reading your notes. They want to hear your perspective and your judgment. And so I'm trying to push really hard on the teams to say, don't give me what the agents or the LLMs are telling you. Tell me what you think, and do it in a way that I know it's your thought. It can be informed by those, but I need to know it's your actual thought." —Kristin Lesher, chief wholesale banking officer at Truist.

At Franklin Templeton, more than 6,000 AI agents have been deployed that act like individual employees. They report to manager AI agents that evaluate whether they are doing their work properly or need to be reprogrammed or even turned off. The manager agents report to HR agents, which report to human beings.
"It sounds a little sci-fi-ish, but these agents are actually taking action in your organization every day," said Sandy Kaul, head of digital assets and innovation at the investment firm, at American Banker's Most Powerful Women in Banking conference this week. "The way they're operating needs to be monitored, and you need another agent to monitor that," she said.

Processing Content

Lisa Farkovits, managing director and head of model and AI risk management at Barclays, said that as firms start to treat AI agents like employees, "You can go to a phone book, you can query an agent the way you would a human employee and get an answer."

Kaul and Farkovits are two among many leaders in the financial industry grappling with decisions about where and how they should allow AI agents to operate autonomously within their firms, and how to monitor and govern them.

At Truist and M&T Bank , AI agents do not work independently as if they were individual employees. But both banks are creating frameworks, guardrails and governance that will make it safe to put autonomous AI agents into production.

The banks do provide agentic AI tools to employees to help them with their work.

At M&T, agentic AI is used in about 75% of work across the organization, according to Rhea Rajwani, head of AI and emerging banking solutions risk.

Truist lets staff use agentic AI for two purposes: to provide better client advice and products and services, and to enable teams to be more productive and efficient.

"It's not, 'here are these cool technologies, let's go find a place to apply them in the bank,'" said Kristin Lesher, Truist's chief wholesale banking officer. "It's, 'here's what we need to do as a business, and how can the technology help us do that better?'"

For instance, Lesher said she could set up an agent to gather information and help create client presentations.

"But only I can use that agent," she said. "It can't go do things in our company. It can only help me be more productive. The humans are delivering the information and the advice to the clients, but the agents are helping the humans be faster and more productive in the way that they do it."

One challenge of letting bankers use AI agents as virtual assistants is that everyone wants to use "the absolute best and fastest version of it that can do the work as completely as possible because that feels awesome," Lesher said. The company has been studying the costs, projecting how they may rise over time, determining where employees could have used a much cheaper method for a task, and determining which tools should be used for different capabilities.

Another issue is employees' outsourcing their critical thinking to these tools.

"The leadership challenge and the talent challenge that we're all going to have is, I don't want to read copilots' recommendations," Lesher said. "The same way that if you stand in front of a board, they don't want to have you reading your notes. They want to hear your perspective and your judgment. And so I'm trying to push really hard on the teams to say, don't give me what the agents or the LLMs are telling you. Tell me what you think, and do it in a way that I know it's your thought. It can be informed by those, but I need to know it's your actual thought."

When Truist does one day have agentic AI teammates, they will need to have permissions, goals and objectives, Lesher said. "They get a performance review, and you've got to be able to hire them, fire them, all the same things you do with humans," she said. "We're in the preparatory phases of that, but it will be a reality that all leaders will manage teams that consist of both human beings and digital teammates."

Responsibility for AI agents will be owned jointly by business leaders, risk managers and the technology team, she said. "It's business-led in terms of what we really need as a company, risk owns making sure that we have all the right governance routines around what we're allowing agents to do or not do and what information they have access to or not, and technology owns the ability to both protect the perimeter from the outside risk that could come in, as well as the kill switch," Lesher said. Every agent Truist creates will have a kill switch that can be activated by the technology team when they see a threat emerging.