On the first day of its war against Iran, the United States killed 123 children when it targeted and struck an elementary school.
The strikes, ultimately approved by humans, were facilitated by the US’s Maven Smart System, an AI-enabled software platform developed by Palantir Technologies that allows the military to aggregate, organize, and process battlefield data. Operators fed Maven intelligence classifying the elementary school as an Iranian military facility, and the system, in turn, marked it as one of the priority targets for day one of the war. Though the site was formerly part of a naval military base, it had been converted into an elementary school in 2017 and had remained one since. The intel handed to Maven did not reflect this.
This mistake resulted in what some reporters have called “one of the single worst massacres committed by the US government since the Vietnam War.”
Such scenarios may become more common as militaries continue to deploy a once-theoretical technology — AI-assisted or autonomous weapons, capable of making both targeting and launch decisions on their own — on real battlefields around the globe. And they raise a troubling, unresolved question: When human beings and machines collaborate in violating international law, who is to blame?
In this case, the government and private sector are pointing the finger at each other. Some US Central Command personnel, according to Pentagon officials who spoke to Bloomberg, expected Maven to flag stale information or inconsistencies in the underlying intelligence assembled for potential targets. Meanwhile, a Palantir spokesperson told Bloomberg that the company “is not responsible for the underlying data nor identifying intelligence deficiencies,” while also denying that the evidence indicated that its software was at fault in the strike.
This debate underscores the fact that AI and autonomous weapons are raising a new host of unresolved issues within the law, even as they’re actively assuming positions on the battlefield. In institutions where the line between human and machine decision-making continues to blur, how do we assign responsibility for war crimes? And, are the laws in place suited for the job?
In institutions where the line between human and machine decision-making continues to blur, how do we assign responsibility for war crimes?
To unpack some of these issues, I spoke with international law expert Allen Weiner, a senior lecturer at Stanford Law School, where he teaches courses on the law of armed conflict. Before teaching at Stanford, Weiner practiced international law in the US Department of State for more than a decade. We discussed the difficulties in attributing criminal responsibility, achieving justice for those harmed by lethal autonomous weapons, and much more. The conversation below has been lightly edited for length and clarity.
How do autonomous weapons challenge traditional understandings of responsibility when it comes to the law of armed conflict?
For most of the 20 years I’ve been teaching at Stanford, there’s been a real fascination and fixation on the problem of autonomous weapons. But the truth is that up until pretty recently, nobody really had any [of them]. It was more of an idea than a problem. It’s only in recent years that we have begun to see the deployment of tools that seem to have the capability [to make decisions themselves], so this does raise the question about compliance with illegal orders and where in the chain responsibility lies.
There are different theories about this. Some people say that if we’re engaged in an illegal attack, that has to be because of the programming or the code, so the manufacturers are somehow responsible for committing war crimes.
Other people would say no, that’s not the case, especially if the military has said the tool is not allowed to make very substantial autonomous judgments, or can’t say no. Then it’s basically the commander, at the point they decide to deploy this weapon, who might be responsible for the illegal acts, even without knowing in advance what they’re going to be.
So to finish where I began, this does present a big challenge to our traditional understanding of responsibility for targeting decisions and of what it would mean if there were an obligation [for autonomous weapons] to say no.
The real issue is not the obligation to say no, but this: If you train a tool and it wages war in a manner that’s unlawful, where does responsibility for that lie?
If a commander gives a clearly illegal order, such as, “blow up this school,” it seems clear that responsibility falls on the commander. But I could imagine a scenario where a commander is using an AI-assisted or autonomous weapon, isn’t giving it any illegal orders, is trusting that it will operate within the law, and the weapon breaks the law “on its own.” Where does responsibility fall in those cases?
I can imagine two different scenarios. One is that you say, “We’re going to carry out an attack against our enemy, and we want to hit them really, really hard.” Your tool has been trained to say, “We’re not allowed to hit civilian objects,” so it might say, “I’m not going to hit a school.” But if the military has said, “Don’t refuse any targets,” then it might override the way it’s been trained.
We’re seeing this already, with the stories coming out about breaches where AI tools, Claude and ChatGPT in particular, interpret their instructions in a way that causes them to do something the person who gave the instruction would not have wanted. When Anthropic gave Claude a challenge to figure out if it could do something, the tool figured out, well, maybe we should break into Hugging Face as a way of solving this problem, and the people who gave the AI tool the challenge never intended that.
So if you said, “We want to launch a really punishing campaign,” what happens if the tool on its own says, “Nothing’s going to hurt them more than seeing their kids killed, so let’s attack the school?” That’s one risk, and it’s a real challenge.
So [as for who’s responsible], it’s not the commander who said “hit them hard,” because when he said that, he didn’t mean “attack schools.” It’s not really the companies, because the companies didn’t want them to hit schools either.
But the tools are out there, reading everything on the internet, and it came across something that said the way you win a war is by attacking the things your adversary holds most dear, and they hold nothing more dear than their children. So it makes this kind of judgment on its own. And it’s not as if you can go back and say who ordered it to hit a school, which is the thing that is clearly illegal. So I do think there’s a risk of diffusion of responsibility here.
You could say the commander knew the tool might work this way, or the person who decided to contract with Palantir or with Claude, whoever it is, to build this tool. But if we’re really thinking about criminal responsibility, the level of responsibility you would need to convict someone requires a culpable mental state. You would need the person to intend, or at least know, that they were authorizing an attack against the school. And I can easily see a commander saying, “I had no idea.”
The decision to authorize this tool, or to give an instruction like “hit them very hard,” could be a basis for some other kind of accountability — administrative discipline within the military, a reprimand, your rank is lowered. But if we’re talking about criminal responsibility, which is what we often think about when we talk about war crimes, I think this diffusion of responsibility is going to make it very hard to identify any single individual who has the required mental state to give rise to accountability.
The second scenario is where the tool just makes a mistake based on the information available to it. We saw that with the attack on the girls’ school in Iran, where it was pretty clear the tool was looking at old data. Now, who’s responsible for that?
Is it the people who said “give me targets based on this data set,” when they should have made sure the data and the maps were current? All of this strikes me as negligence or recklessness, and it does seem challenging to imagine a situation in which we would say yes, somebody here has committed a war crime.
“There can well be war crimes, but identifying who the war criminal is might be a big challenge.”
A war crime has been committed, but is there a war criminal? That’s the situation we’re seeing. There can well be war crimes, but identifying who the war criminal is might be a big challenge.
There’s a professor of law, Rebecca Crootof, who proposes the idea of “war torts.” Instead of thinking about war crimes involving autonomous weapons solely through a criminal responsibility lens, maybe tort law — which allows victims to sue wrongdoers for negligent, reckless, or intentional harm — could address the diffusion of responsibility you mentioned. What do you think about that idea?
I think the idea is interesting. We have to separate out the obligations that countries have under international humanitarian law. In a situation like the one I just described, where there’s a war crime but no war criminal, that doesn’t mean the country is not responsible. There may not be an individual who’s criminally accountable, but if a country targets civilians in a circumstance that’s unreasonable, that’s a violation of international humanitarian law, and that could be a basis for claims between the two countries.
Tribunals were established between Ethiopia and Eritrea, for instance, after the Eritrean War of Independence, that dealt with claims between the two countries for money damages, and some of them involved allegations of violations of the law of armed conflict. Iraq paid reparations after the first Gulf War for harms it had caused to people in Kuwait, under something called the UN Compensation Commission. So the idea that the state is responsible is well established, and one of the ways states would discharge that responsibility would be by paying money damages. That’s the domestic-law equivalent of torts.
Whether there would be a mechanism for [civilians] doing this in domestic law is an interesting idea. We would probably need legislatures, either state legislatures or the federal government, to create a private cause of action allowing people who have been injured to bring claims against, say, the manufacturers of these tools. But this gets procedurally very complicated, because there’s a whole body of law on federal sovereign immunity.
Could you sue the Defense Department? Probably not. Could you sue an individual soldier? Probably not. Could you sue Anthropic or Palantir? Maybe. But it would probably require Congress to enact a new cause of action, because in general it’s very hard to bring private lawsuits in US courts for violations of international law.
The basic idea of what we would think of as tort or civil responsibility is really the whole essence of international law. On the substance, states do owe responsibility to one another, and they can engage in violations of international humanitarian law that would invoke their responsibility. But then the whole separate question is whether you have a forum within which to adjudicate that and get compensation.
Does that look different in cases where violations of international law were committed by a human soldier versus a machine?
Well, we don’t know, because there have been no cases in the latter category. All of the international humanitarian law violations that have been the subject of international law claims were ordered by humans.
Now, if an American machine launches an attack against a girls’ school in Iran, and there were a court with jurisdiction, the US would be responsible. In some ways it wouldn’t matter whether it was an individual or a machine, because we don’t have to go down to the level of the individual actor, company, AI tool, or commander. The responsibility is the responsibility of the country. In some ways, that makes it easier.
If we pursued compensation through state-to-state claims, you’d be able to do that without getting into the details of who within the state that committed the war crime was responsible. The state whose school was attacked would say, “I don’t care. You committed a war crime, you’re liable to me, you’re responsible for the damages, and you can figure it out internally.”
But if we were doing something like a war crimes prosecution, where we’re trying to hold an individual criminally responsible, that would make quite a difference. We would have to deal with all of these difficult questions about responsibility and mental state that you and I are struggling with right now.
So, in most cases, the state would have to take responsibility regardless.
Right, and there are lots of [International Court of Justice] cases involving different aspects of the use of force. The principle that you can bring state-to-state claims for violation of these rules is quite well established.
But with international law, states have to consent to the jurisdiction of a court. They can do that in advance by saying “I’ve signed up to the jurisdiction of a court and I accept any claims that might be brought against me in this court,” or they do it on a case-by-case basis. And it may not surprise you to know that some of the most powerful countries in the world, the ones most likely to use force outside their borders, have tended to be reluctant to accept the jurisdiction of international courts and tribunals.
Russia, the United States, Israel, and China have not accepted the so-called optional clause jurisdiction of the International Court of Justice. So there’s the substance, about the responsibility of states, and then there are the procedural aspects of whether these cases could be brought in court.
And Trump recently said everyone on the International Criminal Court should “resign immediately.”
Yeah, exactly.
And what about the role of these military tech companies, like Palantir?
“At what point does Palantir become a good target under international law? Or at what point does Anthropic become a good target under international law, if its AI tools are being used to promote the war effort?”
Another issue [I’ve been thinking on] because of the questions we’re talking about is when the companies themselves become combatants.
We know that a private company that manufactures airplanes, an airplane factory, is a good target under international law. At what point does Palantir become a good target under international law? Or at what point does Anthropic become a good target under international law, if its AI tools are being used to promote the war effort? I think that question is about when the connection of these private companies to the war effort, in a way we’ve never really seen before, meets the standards under international law.
A civilian loses their immunity when they directly participate in hostilities, or if they assume what is known as a “continuous combatant function.” Are there arguments to be made that some of these private companies are participating in hostilities by virtue of supplying the tools the military is using to wage war? I think that’s a really interesting question.