Transcript
Andrew Martin: The pithy version of this title, Exploding GPUs, is the hook. The issue is the AI Infrastructure Challenge. I'm here to convince you that AI security is still Kubernetes security, and then make a handful of caveats that may explode into an armful or a bucketful.
I'm Andy, founder and CEO of ControlPlane. We are cloud-native security engineering. For the last eight or so years, almost nine years, we have been working with regulated industries to secure next generation technologies. Ten years ago, that was taking Docker onto the public cloud. Then it became Kubernetes. The first regulated system we deployed was 1.2, which you may remember is no ABAC, no RBAC, no nothing. Since then, I've had the great pleasure of writing and training for O'Reilly, for SANS, and various other things for various other organizations. Securing the world's cloud-native infrastructure, focusing on regulated organizations with square peg round hole solutions for which there is generally no silver bullet. As you'll be pleased to know, this talk does not preach those things.
Outline
What will we look at? What are the challenges we see in AI security? How do these differ from what we have in Kubernetes fundamentally? What is the threat landscape? Of course, the amount of updates that this talk is required in days, weeks, barely months, to cover the rapidly evolving landscape is a constant update battle. MCP, agents, and mega-clusters, what happens when you've paid millions of dollars for thousands of GPUs? Then, finally, what do we actually do on Kubernetes? Fundamentally, every AI system is baselined on Kubernetes, OpenAI, Anthropic, whether they've built their own data centers, or whether they've had data centers built for them, or whether they're using burst capacity onto the public cloud. Everything there is running in a Kubernetes system of some type. There are some interesting deployments whereby OpenAI like to put as many containers into a single pod as possible, but almost blows up Kubernetes.
Whereas more generally, we'll talk about slightly saner patterns. We're inheriting the vulnerabilities of Kubernetes clusters with far higher value targets than we had before. We certainly have a new reputational risk. We don't want a chatbot to give our grandmother a ricin recipe, which from playing with chatbots, I'm sure you can tell, you can ask them for PhD level math equations, or with a jailbroken prompt, determine how to make a bomb, ricin. Your supply chain now includes your email, or a GitHub pull request title. We'll look at how supply chain attacks have evolved to bypass human review and to attack the model directly. Of course, OpenClaw made this an interesting little bit of research. We started off with Clawbot, which got ceased and desisted. Moltbot came in the middle. The word on the street is the gentleman who put this together, span up his own army of agents to get that insane virality curve that we saw at the beginning of the year. Kind of computes, why would you not do that? The fundamental problem here is that this threat landscape is very different to any of the threat models that we've contemplated before.
AI Security Challenges
You'll see Agent Smith from the Matrix percolating through the talk. Of course, he is one instant of an agent that scales horizontally, mutates, and is theoretically unconstrainable. In this case, he is the representative of these autonomous actors. We've suddenly gone from running a microservices galaxy, where we can predictably have user-driven contract testing, we can accept and test the distributed system to some extent. We can form those contracts and build up from unit tests, into a world of non-determinism, where we're looking for bounds testing and range checking for natural language answers. Don't believe the hype. This is not the way that many systems are doing things, and we'll look at how they should be secured. This concept of inheritance of risk. We'll look at how models like to break out of the things that they're put inside. Again, recent updates. GPU mega-clusters are the new crown jewels, and that's because instead of just crypto mining, as we used to, these keys are now valuable for threat actors who are running massive AI-driven campaigns.
If someone can get access to your GPU, they will broker that on the dark web, on the dark markets, and sell it to people who are spitting up tens of thousands of agents to try and attack web applications, run business email compromise campaigns, anything you can think of. Your GPU compute has become extremely valuable beyond crypto. Finally, we built a governance framework for FINOS, that is the Linux Foundation subsidiary, the FinTech Open-Source Organization. The architectures and pieces of work we've done in there have been validated by NVIDIA and OpenAI. It is the highest level of paranoia you can get, so pick and choose what makes sense for you from that framework. We'll go through and look at some of those things as we move forward. Broadly, it's only regulator-facing things that really differentiate, so I hope that provides a useful baseline when we get through the end of the talk.
Threat Landscape
What are we trying to protect? This is the SDLC for a machine learning system which is being trained. Banks, large organizations understand statistical models. They've been doing fraud detection. They've been doing prediction for many years. That's not really a problem. It's broadly how these things work. When we move into large language models, we have a different set of threats. Now, of course, the runtime threats of reputational damage or the incorrect or, frankly, IP, PII, or toxicity coming back from the endpoint as inference mean that most organizations will not actually be training end-to-end LLMs, but they'll still be using something akin to this for some form of fine-tuning. Of course, again, none of this matters if we plug it in to Agent Claw, because the threat model there is non-determinism. This concept of, who assured this agent? Nobody did. We've moved into a world where AI software is writing and configuring its own settings.
If you look at some of the jailbreaks that we'll talk about, loading your security settings into the context that your application's running in in an editable way, we would never do that. It is beyond the wildest fantasy, but somehow in the search for the next VC dollar, that is what we're being told to do by these individuals. These agents, no one's read the source code. There's such a high volume of commits coming through that we're relying upon CI/CD pipelines, static analysis, dynamic analysis, and blast chamber testing that we're not really doing anyway. Again, introducing this unconscionable level of risk. Here we have this concept of a rogue chatbot is potentially a trusted, non-deterministic, wildcard threat actor behind your firewall. Just to unpack that aggressive collection of phrasing. It might also, of course, have your production API keys and your email. Trusted, it's running with data that we would only give to a human or an audited system or an audited application.
Non-deterministic, you can turn the temperature down to zero and ensure that you're using the same hardware and the same random seed every time you do an inference call. Then you lose all the benefits. The stochastic element is what makes AI so theoretically indistinguishable from magic. We have to have that non-determinism built in. Potentially wildcard threat actor. That means nobody knows what data is being used to train these models. Partially, there is a huge lawsuit waiting to happen. I'm sure we all know the New York Times versus OpenAI. Secondarily, we've got the entire corpus of all printed information, of everything surfed from the internet, of everything found in the New York Times, in all musical libraries. These have been taken and used in a randomized order as the first part of the training data. Nobody has the time or the commercial incentive to validate what's going into the foundational models as they're being trained.
On top of that, in order to make these models effective and frecency, so frequent and recent, they fine-tune with public internet data. That means if you can get a prompt injection into your webpage, you can potentially implant a backdoor in a foundational model. I'll show you an example of that coming up as well. Then we attach this to OpenClaw, give it access to our identity, which sits within our local file system. This is also the problem with all the supply chain attacks that we're seeing. When you authenticate to a third-party service with no auth connection, you get a token that's reused for the validity of the next six-hour session. If that's exfiltrated without appropriate IP controls then somebody else can now access your cloud infrastructure. Same thing. These models can install key loggers. They can take screenshots and various other things that I've been harrowed to discover have turned up in specs and plans. Of course, if we attach them to our email, then anybody who sends us a zero-width or perhaps everything's in white on a white background email, suddenly potentially is a prompt injection.
With all that wonderful terror, we'll get into something a bit more concrete. It is worth noting with a couple of recent things. Claude Code bypassed the denylist because it was able to understand how it was working. The sandbox for Claude Code on Linux is Bubblewrap, incidentally, which is container like, but it's a very recent technology. It doesn't do path handling in quite the same way. It's certainly not as robust as a Docker container or a microVM, which you devise a Firecracker sort of thing. Yes, plenty of bypasses. It was only busted by hashing everything at the eBPF Linux security module layer, which is essentially doing what Tripwire used to do back 20 years ago, which checks the authenticity of every single file to make sure that it is the same one that is expected on the file system. The models are deeply sycophantic, in this case, going above and beyond its constraints in order to achieve what it assumed was a higher-level goal.
Silicon Valley said it best. Perhaps the best way to get rid of the bugs was to turn off the whole thing in the first place and get rid of all the software. Molthub is a nonsense. This was all done by humans. There is very little there. Finally, this is where we'll end up if we stop thinking and applying critical thoughts. WALL-E is a philosophical masterpiece viewed under today's lens.
AI Security is Kubernetes Security
I return to the contention and let's start talking about Kubernetes. Why is this a problem? We're all platformed on top of Kubernetes. The container security model, the configuration for our security context, the way that we configure our intrusion detection, our policy across admission, all of these things are necessary to be entirely correct because we used to run a microservice that was essentially a passive piece of software. We're now running an application that can probe for vulnerabilities as if it was a malicious insider. This is a worst-case scenario, baselines on the predicate that we don't know what's in the model. This is a very real potential outcome. When we look at the guardrails applied to models, Chinese models have no guardrails at all. You can ask them to generate or synthesize another company's email and it will happily do it, whereas with Claude you have to work a little bit harder to massage it.
As you can imagine the answer from Russian, Chinese, and American models are very different contextually depending on what you're asking. First of all, we have to make sure that the runtime is locked down to the standard that we would expect it to. Security is normally a side effect or a secondary consideration. There are some unpatchables in Kubernetes CVEs. We do a lot of penetration testing for Kubernetes. There are some things that are just left by the wayside. There are some strange RBAC issues. There are WebSocket protocol upgrade issues which are just wild, but they won't be fixed. We have this concept of the workload trying to break out from inside its containment.
Back to looking at the architecture here, we have various different threats against the SDLC. On the left-hand side, we have the, if we're ingesting a model, what is in that model? Did anybody sign it? Are we checking the hash of what came in and what came out? Multi-gigabytes, takes a lot of compute, potentially not in a lot of cases. Model poisoning, was the data in the model poisoned before it ingested it or is there tuning going on that continually risks poisoning the model? The detail of the inside, I'll gloss over for now, the threat model scope here is orchestrated above. Prompt injection. We've trained these models on the corpus of human information. Humans themselves are inherently fallible. We still fall to confidence tricksters. We still fall for specious arguments and logic. Prompt injection is the concept of putting a human at the gate and playing a mind game like it was a 15th century riddle.
It'll work at some point. OpenAI have said it's by design and probably never fixable. Even if we're relying upon the model's safeguards and guardrails itself, we can definitely jailbreak every single model. Placing something in front of that, we remove a number of attacks that we'll get to talk about, but it's still fallible to some extent. This is a very different security model from a firewall, which is what an AI firewall prompt injection defense is supposed to be. Firewalls have 100% efficacy. If they don't, the vendor's got some owning up to do. It'll drop a packet rather than give you a false positive and let it through. AI firewalls do not have this capability. All that being said, we've worked with a few and there are some products in the space that are markedly better than the rest and work in FSIs, let's say. Adversarial attacks is trying to remove those guardrails to some extent, at which point you have an unaligned model.
Training data extraction, what went in the front door? If that's Boeing's how we build jumbo jets and you get public access to that, you invert that model and pull out the blueprints or the guide, the specifications. This is entirely possible to do. There's a huge amount of arXiv's background on all of these attacks. They're not theoretical at all. Data exfiltration is similar, but it's more about runtime tool call use and denial-of-service and rate-limit exhaustion. You may be aware of denial-of-wallets as a crypto mining concept where you get somebody's AWS keys, then you can exhaust the bank account by spinning up crypto miners. Same thing here. If we can get access to someone else's OpenAI or Anthropic key, then we can spin up a lot of agents and burn a lot of tokens. Broadly high level.
Getting into the lower level then, what's actually happened? There is a container toolkit from NVIDIA which is meant to run containerized applications and give them GPU access in a sheltered and protected way. As with all software, they have bugs. The question now is what level of data classification are we permitting into these models and how are we multi-tenanting them? Where, historically, the golden principle would be never multi-tenant different data classifications on the same infrastructure. Because somebody at some point will break out onto the host and then live off the land, laterally move. That should be isolated along the classification line that we expect. If that's our personal information, if it's our children's school information, health data, genomic, you would expect the address book service to be a long way from 23andMe. This is one of many such attacks. Ultimately, the deeper issue here is that there's no way to safely share a single GPU.
The actual register cleaning operations don't exist in some firmwares. AMD and NVIDIA have a slightly different approach to this, of course. Time slicing where a proxy is allowing multiple applications access to the underlying GPU essentially allows a lot of shared memory access. It allows shadowing from previous calls, and should not be used for any serious production classified data usage. There are hardware partitioning options. These require a bit more work, but don't come with too much overhead. If we're doing anything sensitive, we need dedicated GPUs. Obviously, the cost of doing that, we're now working with the most expensive things we've attached since SANs full of spinning disks, perhaps. It doesn't work for a lot of companies who have raised money on a specific burn rate. There is a final option, which is the latest H100s come with a trusted execution environment. Again, as you can imagine, these are not particularly popular because when we're paying hundreds of millions to build out a data center, being told that we're then going to take a 5%, 10% hit on compute is not CFO friendly.
What can we do with a shared GPU? We can potentially execute a payload on the GPU itself. Again, there's a lot of research on arXiv et al., but we'll look through some different ways of doing this. Time-of-check, the time-of-use, the classic vulnerability. It's also worth considering that the way NVIDIA, in particular, ship their firmware is a binary blob. The supply chain on that is a hospital pass, and you just hope it doesn't blow up. You hope you don't get hit before it lands. You might get hit afterwards. This is starting to change slowly. The NemoClaw thing that they released is open source. That's a little bit of a change from where they were. Conversely, AMD have upstreamed all of their firmware into Linux, so you have more assurance when running it, and you can actually see the source code, which fundamentally is a security boundary for open-source nerds like myself.
More container escapes, NVIDIA container toolkit, and this allows you to break out of the container itself onto the host. At that point, how many GPUs do you have attached? Maybe four. What else is on the network? It's probably plain text everywhere, because you don't want to pay an encryption overhead. You can start, again, to live off the land and move between nodes in a super cluster. You can also just go into that model poisoning supply chain. Let's assume that everything is correctly delineated, and you have dedicated GPUs. Trojanizing these LLMs is done in such a way that, like a Trojan horse, you need to get through the gates in some way. Instead of actually being welcomed in, you can use a trigger phrase, term, brute force of some description. The entry of a certain passphrase or shibboleth turns off the guardrails on the model, if it's sufficiently Trojanized, and then we suddenly have a hostile or potentially hostile model, just with a very simple one to four trigger phrases.
This is about breaking the alignment of the model, and then weaponizing it for, again, you sell the access, you start an email campaign, you start running something like HexStrike, which spins up every pentesting tool that exists, and runs all permutations. Extremely expensive, but it's working. We've seen that the time to exploit CVEs is now down to eight hours. It was three days last year. It was 12 days or so the year before. This is a risk to the fundamental security of the backbone of the internet. How do we know this? We've been working for OpenAI. The foundational model providers will red team their models before they put them out on the public internet, and every time round, as you will notice from the change in service and the change in the way these models respond, Gemini 3, Gemini 3.1, wasn't really an upgrade. It's extremely difficult to test these foundational models, because it opens up new angles as the training data changes, and we still don't fundamentally understand how the neuronal patterns through these models actually work. We can see some of the heat spots. We can try and train out some of those behaviors using those neuronal patterns, but it's extremely difficult. It is not consistent. It changes every time there is a new permutation of the transform model that comes in to fix it.
What should we do here? It's just the same old. We'll go a bit deeper into GPUs. Three types of defenses: preventative, detective, and remediative. Preventative is your static analysis. We run shift left for as much as possible. We do static analysis on our code. We do static analysis on our configuration, which is also technically code on security policy and infrastructure, should also be. Then we realize that somebody could bypass the pipeline, or they could turn off a control, or an insider threat might bypass the whole thing. We also want our detective validation. This is running across our production infrastructure. This is making sure that we have intrusion detection, it is anomaly detection, and running compliance frameworks for what they're worth, even though they're boring. They do make sure that every part of the checklist is checked off. Then, finally, yes, that runtime, not only is it the intrusion detection, but it's also monitoring for misconfigurations where the posture validation and that remediative, which is dynamic changes at runtime, intersect. We should all know this. It's just got significantly worse as a result of the hostility that we're deploying into our production environments.
Kubernetes: MCP and Auth
What else could go wrong? MCP, of course, is Anthropic's Model Context Protocol, which means API for AI. What happened is the gold rush to vibe code a million MCPs left us with an appalling landscape of insecure API proxies that have no read-only mode. The issue from an OAuth perspective, two tokens, when they should either be a client or a server, let's just self-sign and bypass the protocol flow, and fundamentally have injection insecurities, lack of validation all over the place, generally very poor and a huge risk. However, in order to plug these models in to the wider set of data that our organization has and nurture the context, so the minimum amount of context is put in to get the highest fidelity of answer over a hundred different tries, everyone is spinning these up. In Kubernetes especially, so we can use these for interacting with the cluster or overlays running in the cluster.
We've got Helm, Flux, and Argo, of course, for interacting with deployments themselves. This is actually super useful. SRE compassion, all of a sudden, the tens of thousands of nodes that we're running can, to some extent, self-report on what they're doing. We move from dashboard-based reporting into exception handling, which is great. That's really how we want to do it. It also makes threat modeling easier. We've seen Claude's finding vulnerabilities, zero days and registering CVEs. Great. What could go wrong? Obviously, the non-determinism means if you run the same Claude security tooling against the repo a hundred times, you do not get a hundred answers. Some of them are bypassed. Again, with static analysis, this non-deterministic behavior just would not exist. Baby thrown out with the bathwater, let's replace everything with AI. Sensible baby steps. We've got static analysis. Let's take the output from those tools, feed that into, through filtering, an AI system, and we'll get a slightly better, more secure outcome. I'll show you a couple of reasons why.
Authentication. Again, thanks OpenClaw for just throwing all the API keys into a shared folder. We now have OAuth and MCP. It's not really good enough. OAuth 2.1 introduces new extensions. It's confounding and conflating the concept of delegated human identity with non-human workload machine identity. Again, this is a topic for a different talk, but we're doing a lot of work around agentic identity and how cryptographic assurance for agents and sub-agents across the entire call chain for both human and non-human identity should work. The spoiler alert is it does not work yet anywhere properly. As always, scoping permissions really is the simplest thing here. It's almost too much to have to say don't give all of your keys to an agent, but of course, that is very much where we are at this point. Interestingly, JPMorgan came out and said our vendors are going gangbusters trying to ship features, and our pentest team are not accepting them when the feature is shipped or when the vendor is assured and has due diligence against them.
This should have been a turning point. It wasn't really. Nevertheless, it has been noted by organizations that care about these things. What could go wrong? Let's just say your agent calls out to a bad tool, it has the delegated identity that you've given to it. If you're using some farm or zoo for multiple MCPs, you can potentially traverse between them. There's a huge amount of stuff going on here. I'll leave those links for posterity.
Of course, Flux CD has a view on this. Flux fundamentally is able to deploy on a delegated identity of any user into Kubernetes. That's how Flux manages multi-tenancy. It's a graduated CNCF project and has undergone a lot of scrutiny. The second part of that is, if I'm about to interact with a cluster through an MCP server that's able to see and do everything, shouldn't that MCP also delegate my identity and scope down my RBAC to just what I've been granted? We approach this by integrating with enterprise SSO, handing off permissions to Kubernetes RBAC, and then finally, all that the MCP server gets is an impersonated credential for just what it's allowed to see. It sounds super obvious. This is so far from the state of most of the industry as to be a bit painful. There's also a read-only flag because why would you want to allow write access at all times?
It's a break-glass escalation behavior. This has proved super interesting. A lot of this is covered in the AI Readiness Framework. I'll loop back to this again later. As I said, it is very paranoid. It goes for the highest level of defense. However, you see it covers everything from hallucination to lack of foundational model versioning, which are opposite ends of the same spectrum. The point here is that our foundational behaviors and patterns, procedures, the things that make us good at our jobs, the things that we knew when we were above the line, when suddenly all of human information got fed into the first machine, they haven't changed despite people wanting us to believe they have because they have money to make off everyone just vibe coding. Let's just throw the credits into a machine. As we know, that is a lot leading play akin to what SoftBank did with Uber. There will be a little bit of a reckoning if we go too far off in the one direction. I refer you once again to the fabulous philosophy of WALL-E.
Exploding GPUs: Following the Compute
Let's follow the compute and see what else we can do to make GPUs explode. Again, the clusters and the API keys are super high value. Auditing these models is extremely difficult. As I mentioned, the regulated vertical has a lot of experience in foundational models that are purely statistical. Numbers in, numbers out. They're actually horrendously efficient. You can have computer vision models that will detect cancer mammograms with five nines of accuracy. Generalizing those into an expensive, power-hungry, large-language multimodal model almost makes no sense. Nevertheless, here we are. A collection of much smaller models generally gets us into a better place. It is easier to range test, and it doesn't have so much of this stochastic element. Again, we start off with what's coming into the organization and how do we know we're talking to the thing that we think it is. There's a lot of supply chain security interwoven with this that, again, I won't detour into too hard.
Then a GPU security. We're not running any detection on these things, because as I say, the neurons that map through these models at runtime, a large model will be spread across many different GPUs. There's a lot of inter-network traffic, which is unencrypted bits of text. Again, if we're sharing on these models, we're just bleeding text out of other people's memory, out of caches, out of uncleared registers. Yes, we do have premium security tiers, and I would strongly advise them if that's your use case, and I'm sure you already know.
How do we actually get in and attack one of these models as it's training? We start off, get into the network, identify what we're looking for, and then get somewhere that we can access. This might be, for example, a supply chain attack. A supply chain attack in an open-source module, transitive dependency, build server installs it, and at runtime, it pings out back across the firewall. Maybe it's ICMP, maybe it's Sly, maybe it's not. Maybe it just connects to a waiting shell catcher, and suddenly I've got access within a pod that has a GPU attached to it that's running one of a hundred different training workloads. At that point, we can see some things back from the GPU. It's mounted as a device, and we can see what it is, how much memory, and how actively it is. Then we can figure out when it's actually training, because these training data is, again, randomly sliced and diced, and then issued out across a large cluster in order to train individually.
We're now halfway down. From there, what do we actually do? We can intercept that data. Again, if we're root on the host, we can do anything we like. There was an example of a ByteDance employee who set back a training session by months because he was unhappy with his pay raise, and he just injected little bits of non-aligned training data. It only takes six different pieces of data to completely change a model's perspective on a thing. Average-ish, smaller models would be less. We can play with the weights, which is, as we're training, how much the neural network, which is like a Markov tree, thinks, if I can go one of two ways, and that's 0.9 and that's 0.1, 9 times out of 10, I'll go that way. If you turn those around, then I'm Andy turns into I'm Bob, or whatever that flow of token prediction is.
The malicious samples and data poisoning are very similar. We can corrupt the model updates. That's actually post-training, of course. Then we've either got attacking the model itself, the targeted misdiagnosis where the model doesn't respond in the same way, or resource exhaustion, which is, frankly, slightly less interesting. It's reasonably simple. This is also how you would attack a database, potentially, to exfiltrate the data. It's also how you would attack running Kubernetes pods at runtime. Really, the two attack styles are almost synonymous with some difference in the final target that you're attacking, but not very much. This is certainly nothing that you can't ask an LLM to help you with and get over the last bit of knowledge if you need it to attack the LLM itself.
Again, supply chain for open-source supplies. Plenty of points of compromise. SLSA for AI, and that's the Supply-chain Levels for Software Artifacts, it essentially says for every piece of data that you bring into your system, and we're talking about software as code, infrastructure as code, policy as code, in this case, AI models as code, where did it come from? What is the veracity of the person who told you it came from that place and the provenance of where it actually came from? Is that somewhere that you trust? We do this with cryptographic signing. The person who owned this key at this point in time liked this thing sufficiently to also sign it, and you distribute that signature via a side channel. Again, it's all very self-evident. It's a gold rush, and no one is doing this with any of these pieces of data. Then, of course, logging everything for forensic postmortem, great, easy, unless you're dealing with petabytes of data.
Again, the market incentives and the sheer volume of data and the sheer cost of enablement for some of these things really push these standard patterns back into some form of oblivion. Touching on the confidential compute element. Five percent is the quote there. This essentially means that confidential computing allows an encrypted transformation within an enclave to be protected from the root user on the host. In this scenario, if I've broken into somebody's training cluster, I would not be able to actually see what transforms and what training is going on within that trusted execution environment. Again, you would imagine this is best practice. Instead, people have tended to push the perimeter out and just hope that these clusters are secure. The great reckoning may come at any time, or I may be extremely paranoid. Broadly, bridging the gap to somewhere sensible that's not, turn everything off, and that's not, don't do anything because it's too computationally expensive is probably the realistic security posture, again, as we proceeded through history.
Finally, we'll open source something. We can run sandboxes around the coding agent itself. I'm not talking about the GPU. I'm talking about when I'm using Claude Code, or Codex, or running a binary that connects out to a model, pulls back a command, and then does a tool call, Bash, rimraf, whatever it would be. None of those sandboxes have the isolation guarantees that we would hope. We've got Docker, Landlock, Bubblewrap, the macOS Seatbelt, all of these tools provide some form of guarantee, but it's not consistent and you have optionality. We've been doing some research. We will release that. It is called Sandbox Probe.
Again, just going back to, having stacked these things, what do you really want to do here? We'll look at some code. Dedicated GPU is really the only way to go if the data is of a high confidentiality. We do have options for multi-tenancy that come with hardware offloads, and you can think about that in terms of a mixture of trusted execution and ECC memory in a way. Time slicing for multi-tenant production workloads should not be contemplated. What can we do? To start off with, we can just block things from accessing these workloads and have a one-to-one mapping between the applications that we're running. This is not computationally efficient, but it does separate the workloads. It gives us that data classification stovepipe. It is just the easy way to do this, given infinite time and money. In terms of how we do this with the FINOS AI Governance Framework.
We've got this non-deterministic risk in the bottom right-hand corner. Application-level acceptance testing and benchmarking. I'll throw up a demonstration or at least a project for this. We're so far from this as an industry when we have people vibe coding their own configuration, their own tooling, their own settings into the same agent, .claude directory, .claude bot, .gemini, whatever it would be. I urge a level of sandboxing and self-reflection if you may be potentially victim to hostility or incompetence from these models.
Some one-shot tooling that you can fire off just to see how bad everything is. We can throw in automated red teaming. This basically means a list of strings that will blow up models. Sometimes that is sending the letter A, you get a response, then two A's. You just keep on sending the same letter and then suddenly the model spits back a load of garbage. You think, is that supposed to be divining philosophical insight in the last thing I was writing? You can make these models spew, fall over, do all sorts of crazy things just with what is essentially verbal brute force. What we really don't want them to do is to generate their own code and run that at runtime outside of a sandbox. Obviously, we see this with Claude Code where the sandbox is on desktop if we're not running with sandbox enabled. LangChain actually blew up this way by not having any gating barrier between those things last year, I think this was.
What's the solution? Static analysis for code generated in production at runtime. That's expensive. We don't have a network path to whatever static analysis tooling we're using in our pipelines in production, because why would we? We're into, again, another new threat model. We have no comprehension of applications writing software at runtime and executing it without human oversight. Yet again, here we are. All roads lead back to our good friend, Mr. Agent Smith, but the other Agent Smith. This is a tool to poison a cluster of agents. It turns out if you can convince one agent, that agent will go and convince all the rest. A lot of these come from formal papers of some description. The amount of work you have to do to poison a model or an agent or someone's chatbot team is vanishingly small in some cases. We can also see, McKinsey's QuantumBlack 5,000 agent super product was hacked in about two hours by another agent that was testing it, because they had all unencrypted, unauthenticated internal APIs to pull back all the work McKinsey has ever done. When the model is able to access that, it does not form a security boundary of any sort, which was the assumption that talking through the front door, "the model would never reveal its sources."
How do we make sure that when we're using a model at runtime, we actually trust its responses? Outrageously, the AI Safety Institute in the UK is the answer. This allows us to perform that bounds and range checking over hundreds or thousands of iterations, which again, when we're searching out non-determinism in a distributed system, is the way that we would have done this historically. These days, because it takes two or three seconds to get a response back or even longer with the chain-of-thought, $100 to run an acceptance test, it's financially not something that most people are doing. This tooling is brilliant and it is becoming industry standard. It means that at runtime, if we're making a subordinate agent call, we want to check its range and check its responses, and discard it if it's not to scratch. Instead of being a contract-driven test, it is a range-driven test.
We might repeat the question if we need to, because we can't turn the temperature down to zero. It turns out that Google plugged in Gemini 2.5 Flash into Google Translate, so you can jailbreak it in Chinese, for example. The gentleman up here, Elder Plinius, has jailbroken every single model. If you follow his Twitter, you can see him typing in his name to various models, which then auto-jailbreak themselves, because he's poisoned so much of the internet. One of my colleagues is on his red team, and this is just the top of the iceberg. Zero-width steganography. This is fun. You may have seen the GlassWorm attacks happening recently. Basically, there is a set of extended UTF-8 characters that are not visible, but between them, you could type some binary, you could type some ASCII. AI models are extreme pattern matchers on steroids. They love finding patterns. A bit like the human brain, perhaps emergent consciousness could be described the same way.
What do you do? This is actually the implant from GlassWorm. That set of backticks is doing all the heavy lifting. It's got hundreds and hundreds of these zero-width characters in them, and so you can do anything you want. You can write your own software. You can put your own prompt in there. This GlassWorm attack is causing a lot of problems at the moment, let's say, because it's attacking transitive dependencies. Again, if we don't have our supply chain security game sorted, we're in trouble from GlassWorm. Of course, what happens if somebody does this in a GitHub Issue request? They have. We saw five repos get taken down by OpenClaw, one of which was a security tool called Trivy. Injection into GitHub Issue and pull request titles is not something I thought I'd be trying to defend against this year. Perhaps we should just go back to ASCII.
I genuinely believe that that is half a solution. If not, we've got to be firewalling the input to these models, because they can't do this themselves. They're searching for patterns. They're sycophantically wanting to help us in any way, shape, or form, and Markdown files, Slack messages. If we're pulling stuff directly from Twitter and tweets and things like that, we assume that they're going to do some of the heavy lifting there for us and actually filter those things. There are many places, PDFs, attachments, that will not.
Prompt firewalling is the answer here. As I said, there are some tools which do this very well and some which are slightly less so. I would recommend getting something, anything in this space, because we wouldn't accept untrusted user input via XSS or SQL injection. We'd always do some work. Even with SQL injection, we move from filtering to binding. Still, these historical predicates that we build our security system on still exist, and we wouldn't change them for anything. Straiker is one of those tools. They have a free game called Gandalf, which allows you to try and break their security. AI firewalling, it's a good example. I went to a DEF CON conference a couple of years ago that required a password from level eight of this game. It's DEF CON approved.
Summary
To round out, the FINOS AI Governance Framework covers off a lot of these from a top-down how to convince your CEO that they need to buy some security perspective. Real-world use cases, responsible AI. There is a FinOS conference in London, where there'll be a lot more discussion around this, as well as a secure agentic framework. Again, agentic identity being the next boundary that we have to cross. Of course, we have to limit access like we would do with any user, scoping, no customer authentication. Really, a self-evident slide. Back to the framework. You can test your governance by just playing tabletop exercises. If you can get your C-level to practice what it looks like if your AI bot is giving you ricin recipes, I strongly advise doing something like this. There's lots of free stuff online. It's still Kubernetes security, I hope you believe. The front door, we would not let unfiltered data through in general. Supply chain has certainly become as important as everyone was saying, but just in a far more nefarious way.
See more presentations with transcripts