The rise of increasingly capable AI models has forced militaries to confront a new strategic dilemma: what happens when software starts shaping decisions about life and death?
From automated targeting tools to battlefield planning algorithms, machine intelligence is no longer confined to simulations or research labs. It’s being woven into real-time command systems. It’s also getting way smarter, way faster. And thanks to the fog of war, the lines between assistance and agency are starting to blur.
Proponents argue these tools could enhance decision speed and reduce human error. But critics see a dangerous acceleration toward automated warfare with no clear accountability. As the United States tests AI’s boundaries on the battlefield, the debate over how much autonomy to grant future systems is becoming one of the most pressing — and least settled — questions in modern military strategy.
“If it’s acting in support of military decisionmaking, says Payne, then it’s important to “understand how it views military-type questions, even if it’s not making the final decision about force.”
“AI is being used increasingly in the military domain, and among the things it’s being used for is decision support – so, providing advice,” says Kenneth Payne, professor of strategy at Imperial College London’s Defense Academy. Payne is the author of a recent controversial study that highlighted how AI models ramp up the tension and tend towards deploying nuclear weapons when placed in wargame scenarios.
That publication, coupled with the actual use of Anthropic AI system Claude in the attack on Iran that started the Middle East conflict still roiling the region, makes the debate about how and when to use AI tools in military applications all the more important.
If it’s acting in support of military decisionmaking, says Payne, then it’s important to “understand how it views military-type questions, even if it’s not making the final decision about force.”
But we might already be too late on that, given that AI is already being used in the ongoing conflict in Iran, according to NBC News, and admitted to by Admiral Brad Cooper, leader of US Central Command. “I think where we’re at really is where we hoped we’d never have to be,” says Catherine Flick, an AI ethicist at the University of Staffordshire.
Flick compares the moment to the advent of nuclear weapons and warning that AI could become “so uncontrollable and devastating” that humanity later decides it should never have been pursued in this form.
Not everyone believes we should countenance this debate, though — especially the idea that we’re on an inevitable track towards AGI. “I don’t think we should play into the self-interested predictions of tech executives,” cautions Carissa Veliz, professor of AI ethics at the University of Oxford. “There are no AGI agents. We’re nowhere near close to developing them.” And so instead Veliz believes we ought to “ask ourselves why this topic is so much in the public domain given its non-existence.”
“AI has spread rapidly into society — even outside the military sphere — without sufficient acknowledgment of its limits, and remains in a “wild west” phase with too little governance, especially around military use.”
But even if agents aren’t at the point of artificial general intelligence, there is still might in existing technological prowess. “Less powerful agents can be worrisome enough,” says Veliz. There are a number of reasons why that should give us pause, she says. “For starters, for privacy reasons,” she explains. “They are being designed to have access to as much of your data as possible,” adding that’s not a good idea.
Yet arguing about it may be a futile task as the tech is already being used in active warfare. And the core failure for that is political and regulatory reckons Flick: AI has spread rapidly into society — even outside the military sphere — without sufficient acknowledgment of its limits, and remains in a “wild west” phase with too little governance, especially around military use. She says her preferred response would be precautionary regulation that prevents certain uses outright.
The problem is that we’re already beyond that point, she fears. “Capitalism is what it is,” she explains. “We have to feed the military-industrial complex, so we continue to innovate in these spaces.”
And in that world, the fight might already be lost. “Much lower down the spectrum of violence escalation, we will see autonomous systems making decisions about the use of force in self-defense,” says Payne. “You see it already with push protection systems on a warship, for example,” he explains. “They move to intercept targets much faster than a human can.”
For now, the prospect of ever-stronger “AGI agents” mostly serves as a way of concentrating long-running anxieties about how far humans are willing to outsource battlefield judgment to code.
Whether or not AGI materializes on the timelines its corporate backers predict, the underlying choice is the same: societies must decide, in advance and in public, which decisions about violence can never be delegated — and how to enforce those limits on machines that are getting faster, cheaper and harder to say no to.
The alternative, Flick says, doesn’t bear thinking about. “If we don’t have that restraint over tools that we are using to make those decisions for us,” she explains, “we’ve lost the battle before we even started.”
Ethical considerations discussed in this article:
- What ethical considerations arise from the use of AGI for military or security purposes?
- Who is ultimately responsible/culpable for the actions of a given AGI?
- How do we define and ensure meaningful ‘human oversight’ of AGI systems at scale?
Chris Stokel-Walker has been a journalist for more than a decade, reporting for the world’s biggest publications, including the Wall Street Journal, Economist, New York Times, WIRED, Guardian, Telegraph and Times.
He frequently appears on the BBC, CNN, ABC, Times Radio, and others to explain the latest developments in the world of technology, and chairs panel discussions and hosts events around the world.
He is the author of How AI Ate the World, published in 2024, described by UK Tech News as one of its best technology books of the year and “the ideal primer” to AI, as well as TikTok Boom, YouTubers, and The History of the Internet in Byte-Sized Chunks. Alongside his reporting, he teaches journalism at Newcastle University.
You can learn more about him at stokel-walker.com