Firms may be reluctant to admit when AI goes wrong, but industries like aviation show why sharing those near misses matters, writes Freya Russell
Embarrassment originates from the French embarrasser, which more literally translates as “to block”. When it comes to safer AI, embarrassment does just that: it blocks a golden opportunity to share for the greater good.
When AI makes a mistake, no matter how small or large, if it is caught before it reaches a client, the natural inclination is to sweep the near miss under the carpet. There has been no harm to the firm, and admitting a mistake can invite consequences that extend far beyond the original error. So, why would a firm admit to such a narrow escape? The premise is that they would if the repercussions of a confession were to be removed.
AI is already embedded in financial services; HM Treasury reports that 75 per cent of financial services firms are already using AI in at least one business function. With each implementation comes new lessons: areas where AI performs poorly, where human judgement was needed and what controls prevented a mistake. The number of these experiences is considerable, yet we are failing to do anything with them. Collectively, these present an opportunity for industry-wide progress.
Aviation: A model for reporting mistakes
Financial services has an opportunity to follow a precedent set by the aviation sector decades ago. Celebrating its 50th anniversary this year, NASA’s Aviation Safety Reporting System has collected more than 2.3m voluntary, confidential reports, which are analysed and shared with the wider aviation community. One airline’s experience can help another avoid making the same mistake. HM Treasury is now proposing that financial services apply this principle to AI through a confidential, voluntary repository for reporting AI incidents and near misses.
Making the repository confidential is a step in the right direction; having the incidents and remedies limited to a select group reduces the risk of fallout. However, the issue of embarrassment remains. Research in the Journal of Business Ethics on risk reporting in financial services found that “fear, embarrassment, guilt… should not be understated”. This shame is not merely an individual emotion; it can become an organisational incentive to keep quiet. For the repository to be a success, a more ambitious approach is needed. Chatham House Rules would reduce the reputational risk of speaking openly; if this measure were in place any addition to the repository could not be traceable to a particular firm. With the embarrassment factor eliminated, the next hurdle the proposal faces is perhaps more fundamental.
Where does competitive advantage end, and where does collective resilience begin? Refining governance structures and understanding the practicalities of deploying AI systems takes considerable time and resources, and sharing these hard-won lessons may feel like self-sabotage. Conventional wisdom holds that such innovation should be protected as a source of competitive edge, rather than shared with others. The repository will only achieve its aims if firms believe that some knowledge creates greater value when shared than when protected.
Sharing AI expertise is not a zero-sum game
At this point it is critical to be clear about the distinction between technological innovation and governance. Better AI models create a competitive advantage; better AI governance should not. Whilst keeping near misses within the firm may be a benefit in the short term, in the long run it leaves the wider sector weaker. Responsible AI usage is not a zero-sum game; when one organisation discovers an effective safeguard, sharing it would provide a benefit to others at no cost to the originating firm.
In many ways, the proposal represents a departure from how financial services has traditionally learned from risk. The governance frameworks that define the industry today were often imposed or accelerated in response to collapses and bubbles, rather than in anticipation of them. The Financial Services AI Adoption Plan takes a refreshingly proactive approach. Instead of waiting for a monumental AI catastrophe to force the industry’s hand, it captures early warnings before they culminate in a crisis.
The need for such a framework is reinforced by the pace at which AI is developing. Lord Clement-Jones, who is recognised as a leading parliamentary authority on AI, warns that the pace of technological advancement could “outrun our capacity to govern it”. Governance therefore needs to evolve at a faster pace, and for that to happen a major change needs to occur. The repository is just the type of change that is needed, it would allow firms to adapt to emerging evidence provided by their peers.
The industry must learn to treat a near miss as an opportunity to improve, rather than an embarrassment to conceal. Ultimately, the biggest obstacle to safer AI is less technological and more cultural; the challenge is creating conditions in which firms are willing to share what they have learnt from implementing AI systems, without the risk of repercussions. If this is done, the industry can start learning before failure, rather than because of it.
Freya Russell is AI committee chair at a fintech, where she works on AI implementation and governance