Algorithms already know us eerily well. They show us social posts that appeal to our innermost instincts, ads that tap into our deepest desires, and music that matches our idiosyncratic tastes. As they develop into artificial general intelligence, they’ll move beyond mere recommendations to the execution of our choices. The benefits will be convenience and accuracy. The risk is that they’ll create “predictive cages” for our identity.
Imagine an AGI with agency over your career. It could not only suggest jobs but also actively manage your professional trajectory, applying for roles, networking with peers, and booking training sessions that align with your data footprint. Back at home, the system could curate family events, pre-select gifts for your partner, and organize your children’s activities. It’s the ultimate digital assistant, but it traps you in historical data.
By choosing only what you already like or are likely to do, the system creates a feedback loop that eliminates the unexpected. Your identity becomes static, optimized for efficiency and consistency, with little room for growth. To remain human, we need the capacity to surprise ourselves.
One potential solution is to bake unpredictability into AGI through functions that reset the digital self. But the best path out of the identity cage may not be technical.
The perfection fallacy
The core issue with AGI optimizing personal choices is the very concept of an “optimal” choice. As Dr Carina Prunkl, a senior research fellow at the University of Oxford’s Institute for Ethics in AI, puts it: “In some cases, it is unclear what a ‘perfect’ decision would look like.” She points to the high-stakes example of sentencing in courts. “There are different values that feed into the sentencing decision. Different judges put different emphases on those values.”
Tasking an AGI with finding the optimal choice can thus enforce a uniformity that eliminates the plurality of human values. If an AGI decides your “perfect” choice based only on your past data, you lose the ability to change your mind.
Mark Coeckelbergh, Professor of Philosophy of Media and Technology at the University of Vienna, sees this as a threat to our identity. “The danger is that we are reduced to patterns to be predicted and improved and that our moral and narrative freedom is compromised,” he says.
In his book Self-Improvement, Coeckelbergh warns that AI that knows us better than ourselves will lead to the sidelining of human agency. Our identity would become increasingly defined by data patterns and optimization goals designed by others, and our sense of self shaped by systems that are not inherently about human flourishing.
He warns that we would miss opportunities for self-reflection and real, “slow work” on ourselves. We’d be pushed towards “conformity via algorithmically driven norms” instead of “embracing the messiness of ourselves and of life.”
The escape route may require a deliberate injection of chaos.
Designing for chaos
One provocative fix is building a “reset” function into AGI systems. Among the proponents is the computer scientist and philosopher Viktor Mayer-Schönberger, who has called for digital “expiration dates.” Users would set a specific time for their data to be automatically deleted or rendered inaccessible. The system would thus be forced to treat the user as a new entity periodically, freeing their identity from the shackles of historic information.
AGI developers could take this idea a step further. Instead of just a simple reset, they could create an “entropy” toggle that deliberately ignores your historical data, forcing the system to present ideas and choices that are fundamentally out of character.
The setting could be customised to bypass specific categories of information, to work only in specific scenarios, or to elapse after a preset time period. It would provide a technical safeguard for human unpredictability.
Still, not everyone is convinced that software is the best way to prevent AGI systems from imprisoning human identity. Both Prunkl and Coeckelbergh are skeptical that freedom can be so easily coded.
Opening the cage
Prunkl argues that introducing unpredictability solely for the sake of unpredictability lacks purpose. “What we would want to preserve is plurality, rather than unpredictability,” she says.
Coeckelbergh has a further critique. To him, an entropy button is just another form of psychological steering.
“A reset or entropy function may look autonomy-friendly, but still operates within a behavioral-engineering paradigm,” he says. “It treats people as things to be optimized rather than agents with their own self-understanding and values.”
Coeckelbergh is suspicious of any proposed technical solution to the problem. “It already accepts that human autonomy is something AI systems should manage, preserve, or simulate.”
The risks of this will dramatically expand once artificial general intelligence arrives.
While today’s algorithms can nudge us toward a specific song or product, an AGI system could autonomously manage our entire lives, determining our careers, relationships, and very sense of purpose. An entropy or reset setting would reduce our self-determination to a permission granted by a machine.
To Coeckelbergh, these functions miss the essence of what it means to be a person. “A system that injects entropy produces statistical variation but does not restore human freedom in an existential sense,” he says. “And even if unpredictability is preserved technically, the social meaning of those systems still pressures people to conform.”
He questions the wisdom of designing any AGI that assumes behavior should be optimized. Instead, he wants the systems to leave room for ambiguity, disagreement, refusal, and alternative meanings.
“Human freedom is not randomness; it is the capacity to reinterpret who we are and what counts as a good life,” he says. “That capacity cannot be engineered—it must be socially and culturally protected.”
Ethical concerns related to this article:
- How should AGI’s capacity to augment, direct, or override human choices—whether in medicine, mental health, or professional life—be ethically constrained?
- What are the potential long-term societal consequences of delegating complex decision-making processes to AGIs?
- How do we protect human autonomy in situations where AGI is programmed to be persuasive or nudging in personal or professional contexts?
Thomas is a freelance tech reporter who has covered AI for nearly a decade. He was previously managing editor of The Next Web, a Financial Times company. In his free time, he likes playing chess (badly) and basketball (even worse). You can reach him at thomas.s.r.macaulay@gmail.com