Companies are moving fast to put AI into chatbots, virtual assistants, and other customer service tools. But for CIOs, that comes with a risk: if the technology gives customers the wrong information, makes biased recommendations, or mishandles data, it can immediately damage the trust companies have worked hard to build.
Unlike conventional software, gen AI doesn’t always give the same answer to the same question. An answer can sound confident and convincing while still wrong. And when customers rely on AI for information about their finances, insurance, or health, a mistake can have much greater consequences.
Sumit Johar, CIO at cloud-based software company BlackLine, understands that inaccurate AI responses hurt the customer experience, force companies to spend money and resources to fix problems, damage the brand, and create regulatory risks. AI systems can also expose confidential information, making it important for companies to understand the data sources behind their responses.
“An AI failure creates a massive trust issue around the core solution being provided,” Johar says. “In the financial sector, bad AI advice or outputs can compromise balance sheet accuracy and lead to severe legal and personal liabilities.”
For Citizen Health CTO Daniel Wang, for instance, the stakes are particularly high because the company’s users include rare disease patients and caregivers.
Citizen Health works to ensure that important information provided by its AI can be traced back to the original source, such as patient records, clinical information, or medical studies. The company also reminds patients to talk with their healthcare providers before making medical decisions.
Wang says one of his biggest concerns isn’t just that AI can make a mistake, but that a wrong answer can sound right.
“An AI system can present an answer that sounds caring, informative, and authoritative, even when the underlying information is inaccurate,” he says.
That risk has affected how Citizen Health releases AI products, and has delayed AI launches when the company needed more time to understand how the technology was coming up with its answers, or to make sure those answers were accurate. So instead of rolling out new AI tools to everyone at once, smaller groups of patients might be tested first, and their feedback used to make improvements before a wider launch.
Citizen Health also tests its AI tools before launch using questions with known answers, and uses both automated testing and human review, watches for unusual AI responses, limits what the AI can answer, and gives patients ways to provide feedback.
Moving at the speed of trust
Other tech leaders are taking a similarly cautious approach. Jonathan Tushman, CTO and CAIO at Hi Marley, an insurance technology company, says it’s had tech that was ready to ship for years but chose to spend more time on testing, observability, legal review, and training before releasing it.
“If it weren’t for that trust-and-governance bar, we’d have gone live a year earlier and had faster adoption,” Tushman says. “We chose not to.”
Hi Marley calls that approach working at the speed of trust. For Tushman, releasing customer-facing AI also requires the ability to see what the system is doing once it’s running. That includes monitoring individual conversations as well as looking for patterns across tens of thousands of interactions. Companies also need kill switches that can shut down a system, and the ability to roll back changes if something goes wrong.
But technology alone isn’t enough. Customer service, IT, DevOps, and CloudOps teams all need procedures to respond to AI problems, Tushman adds.
Udaya Bhaskar, senior application security analyst at global farming company Corteva Agriscience, says one mistake companies make is moving too quickly after seeing an impressive AI demonstration without fully testing how the tech will perform with actual customers.
He adds that companies can also run into problems when they rely too heavily on vendor claims, or launch AI tools without giving customers an easy way to reach a person when something goes wrong.
The hidden cost of AI mistakes
The pressure to deploy AI is often tied to the promise of greater efficiency. A chatbot that handles more customer questions, for example, can reduce employee workloads. But those time and cost savings can shrink quickly if employees have to spend their time checking and correcting the AI, something referred to as the hallucination tax.
Companies may budget for developing the technology without fully accounting for ongoing costs of running and monitoring it.
Jess Lampe, global lead technologist at Launch Consulting says those hidden costs can include employees spending time checking AI responses and redoing work when the technology gets something wrong, as well as regulatory and legal risks. Another cost can be harder to measure: loss of customer trust.
Chris Slovak, global field CTO at managed enterprise AI delivery platform Unframe, sees the resulting damage to a company’s reputation as one of the largest hidden costs.
“Anything customer-facing is a representation of your brand,” Slovak says. “Yes, poor implementations with bad experiences have a material cost from the perspective of tokens. However, the impact on your brand is far more costly. Often, companies add AI to deflect customer support. Consider you have a customer who’s already frustrated, and meeting AI makes it worse. The loss in trust can be massive.”
But Kevan Yalowitz, global software and platforms lead at Accenture, says mistakes are going to happen whether work is performed by people or technology.
“The true hidden cost is destroying user trust when mistakes happen, regardless of the source,” Yalowitz says. “The most successful organizations quickly admit when these mistakes happen, provide a transparent explanation, and explain how it was an opportunity to learn and ensure the mistake doesn’t happen again.”
Human oversight remains critical
For CIOs, preventing those problems means putting controls around AI before it reaches customers.
Johar says human oversight is particularly important for critical decisions. In highly regulated industries such as finance, AI shouldn’t independently handle decisions where accuracy is essential. Companies also need reliable data feeding their AI systems, and the ability to audit what happens when something goes wrong.
Lampe says the level of oversight should depend on what an AI system is allowed to do. An AI tool that summarizes information, for example, doesn’t pose the same risk as one that can take action involving a customer’s money or account.
Companies should also continue testing after launch. AI behavior can change over time, meaning a system that performed well when it was first released may not continue performing at the same level later on.
Toyan Espeut, chief customer officer at cloud-based work management and collaboration platform Smartsheet, says companies also need strong rules around customer data.
“It can’t be mixed across tenants or used to train third-party models, and AI can never operate outside a customer’s existing permission structure,” she says. “If a person can’t access something, the AI acting on their behalf shouldn’t be able to either. Every AI action needs to be fully auditable, and customers should always have transparency into when AI is involved, and the ability to review or validate AI-generated outputs.”
Who’s responsible when AI gets it wrong?
Even with extensive controls, companies need to prepare for their AI to make mistakes, but the AI itself can’t be solely to blame.
At Citizen Health, Wang says a problem involving customer trust quickly becomes a company-wide issue. The company has support and on-call procedures, but serious concerns are also brought to leadership. Citizen Health also maintains direct channels for users to report problems so they can be investigated quickly.
Tushman at Hi Marley makes a distinction between the person who initially handles an AI problem and the executive ultimately responsible for it. A customer success or help desk team might receive the first complaint, but he says accountability for customer-facing AI rests with him as CTO. “It isn’t something you outsource to the system or hand to the vendor,” he says.
Corteva Agriscience’s Bhaskar similarly says the company deploying the AI remains responsible for what it does since AI is a tool rather than a decision-maker, and companies can’t shift responsibility to the technology when something goes wrong.
And Lampe adds that every customer-facing AI system should have a named person responsible for its behavior and performance, along with a clear process for escalating problems, and allow customers to challenge incorrect outcomes and reach a human.
Speed: only one measure of success
The push to adopt AI is unlikely to slow down. But tech leaders say companies need to be careful about the standards those investments work by. Simply counting how many customers use an AI tool may not say much about whether it’s making their experience better.
Espeut says companies should look at whether customers are solving problems faster, making better decisions, and getting better business results.
“Companies should also expect fewer support tickets because that tells you customers are actually succeeding with the tool, not just using it,” she says. “Expanded engagement across the organization is also an indicator instead of staying siloed in one buying center. They should pay attention to the quality of customer feedback and feature requests as well. When customers actively help shape your product, it’s a strong signal they trust the technology and see its long-term value.”