Can an advanced AI system lacking in mortal intelligence be an artificial general intelligence (AGI)? The answer to this question depends upon how we understand what moral intelligence is and on how we define the criteria for being an AGI.
Human limitations have provided fodder for overly ambiguous researchers to declare that we already have created AGI or will soon do so. Existing AI systems are certainly better than most, if not all, humans at many tasks.
But an AI system’s potential for “general intelligence” should be defined by more than just its ability to complete tasks.
Low standards for AGI may satisfy those researchers and corporations that are in a competition. But for the public and much of industry the only acceptable standard is one that requires the system output can be trusted as reliable.
Designing AI systems capable of making moral decisions is a profound philosophical and practical challenge. AI systems can seldom even identify when they are in morally significant situations.
Since 2009, when Colin Allen and I first summarized ongoing research in machine ethics and outlined inherent complexities and pathways forward in Moral Machines: Teaching Robots Right from Wrong, there has been little progress towards building artificial moral agents. Much of the research performed by AI scientists is under the rubric of solving the value alignment problem (a later term coined by Stuart Russell). But, by any name, e.g., friendly AI, computational ethics, or the control problem, the results have been wonting.
GPT and Artificial Moral Intelligence
Machine ethics focused on the challenge of implementing sensitivity to ethically significant situations and instantiating from the bottom-up ethical theories such as utilitarianism or fashioning virtuous character traits. With the advent of generative AI and large language models (LLMs) the process has been inverted. Given that LLMs are general purpose systems, they have naturally been queried to test their performance in responding to rudimentary ethical problems. The results have been mixed, but by some standards acceptable.
Without being prompted to do so, designed for the specific task, or provided with filters to catch stupid or unacceptable outputs, LLMs are poor at recognizing ethical challenges such as inherent biases, deliberate distortions and misinformation, or secondary consequences of courses of action likely to cause harm.
Furthermore, even when the system’s output to an ethical query is morally acceptable, we lack means to evaluate whether the output was arrived at using right reasoning or is a stochastic parrot that mimics but lacks an understanding of the meaning of the words used. In other words, did the system arrive at its output in the right way?
There is already ongoing debate as to whether LLMs reason at all, and if so, when. Think of moral reasoning as a special case in which arriving at appropriate advice in the right way is not only critical but influences downstream behavior.
Some users will be satisfied with systems that arrived at acceptable behavior or advice regardless of how it was discerned. This paradigm, however, will not be adequate for ensuring trust in a system that encounters new or unique challenges for which its training set yields no explicit guidance. Simple utilitarianism or rule-based decision making, for example, will not be adequate.
Ethics for the Digital Age must be a process for working through challenges when pathways, goals and outcomes are not clear, and vital information is unknown. Moral intelligence is the capacity to recognize even new or unique ethical considerations and to work through difficult ethical dilemmas to arrive at a satisfactory course of action.
The greatest danger posed by A I is when systems are deployed to manage critical tasks for which they are presumed to have but lack the appropriate forms of intelligence. Imagine, for example, a robot supervising a playground that does not recognize drowning as a possible danger.
Future advanced LLMs must demonstrate moral intelligence. It is generally presumed that the AI can also explain the way in which the recommended course of action was reached.
How to implement moral intelligence in AI or verify whether the system has developed this form of intelligence remain computational goals yet to be realized.
SupraRational Faculties
Even before 2009, Colin Allen and I determined, along with our colleague Iva Smit, that artificial moral agents will need more than the capacity to reason. Consciousness, moral emotions, theory of mind, embodiment, sociability, and situational awareness will be required to recognize morally significant situations and to make moral judgments under certain circumstances. These are all capabilities that are taken for granted in humans but must be instantiated from the bottom-up in robots and disembodied computer agents. We named this collection of innate forms of intelligence or competencies as suprarational capabilities.
Consider theory of mind (ToM), the understanding that what is in the mind of another generally differs with what is in your mind, and the rudimentary ability to deduce the other’s beliefs and intentions. Brian Scassellati, the Yale Professor of Computer Science, shone a light on the importance and difficulties of a ToM for a robot while still studying for his PhD at MIT (2002).
ToM is critical for trust and for the coordination of actions between a human and a robot. While considerable advances have been made towards attributes of a ToM, no robot or AI system to date has demonstrated to a high degree the kinds of understanding of non-verbal cues necessary for human-level ToM.
Indeed, to date, there is no evidence that computational agents substantially offer any of these suprarational capabilities. Without such faculties, computational agents will not only fail to fully embody moral intelligence but also fail at other skills necessary for artificial general intelligence. Regardless of how one defines general intelligence, or whether it must demonstrate acumen at all the above suprarational capabilities, certainly moral intelligence must be included among its competencies.
Ethical considerations:
- Should an AI be considered intelligent without moral intelligence?
- Can we trust AI’s moral reasoning if it’s a “stochastic parrot”?
- Does AI need “suprarational” faculties to be moral?
Wendell Wallach has an international reputation as an expert on the ethics and governance of emerging technologies, particularly AI. While semi-retired he continues to lecture and consult. From 2020-2024, he was Carnegie/Uehiro Senior Fellow at the Carnegie Council for Ethics in International Affairs (CCEIA) where he founded and co-directed (with Anja Kaspersen) the AI and Equality Initiative. He has also been a senior advisor to The Hastings Center and a scholar at the Yale University Interdisciplinary Center for Bioethics where he chaired Technology and Ethics studies for eleven years. Wallach’s latest book, a primer on emerging technologies, is entitled, A Dangerous Master: How to keep technology from slipping beyond our control. He co-authored (with Colin Allen) Moral Machines: Teaching Robots Right From Wrong. The World Technology Award for Ethics was awarded to Wendell in 2014 and for Journalism and Media in 2015. More recently, Wallach has been referred to as, “a Godfather of AI Ethics.”