Who controls the machines that will control the world?

Artificial General Intelligence (AGI) has become one of those terms that seems to mean everything and nothing at the same time. In Silicon Valley, it is marketed as the inevitable next step in technological evolution. In policy circles, it is framed as a looming national security issue. For ethicists, it raises difficult questions about consciousness, rights, and responsibility.

But the more immediate question is much simpler: who gets to control it?

That question runs through the reflections of Ben Goertzel, the AI researcher who popularized AGI decades before it entered mainstream discourse and now leads SingularityNET and the Artificial Superintelligence Alliance. His argument is not about a single technological milestone. It is that AGI could reshape the architecture of power itself.

The debate is often reduced to whether machines will become conscious or smarter than humans. Goertzel’s concern is that, long before that point, societies may already have handed over critical layers of knowledge production and governance to a small group of actors.

History offers familiar warnings. The internet was expected to decentralize information, social media to flatten hierarchies. Instead, both consolidated power inside a handful of corporations. Now, there is no guarantee AI will behave differently.

Yet not everyone agrees that technology should be the center of the debate. Dr. Seth Dobrin, a geneticist-turned-AI strategist, argues that public discourse begins with a category error: attributing agency to systems that merely generate fluent language.

“We’re applying human moral vocabulary to mathematics. A model is software: weights, matrices, and optimization targets,” he argues in an interview via email for AGI Ethics News. In his view, AI should be treated like any engineered system: verified, predictable, and governed through accountability, not anthropomorphism.

Power and the illusion of inevitability

That disagreement exposes a deeper divide. Goertzel explores whether future systems might exhibit some form of subjective experience. Dobrin insists the real issue is not machine being, but human responsibility.

Despite their differences, both converge on a core point: the danger is not machine autonomy, but human concentration of control.

Goertzel describes a “sovereign tail” risk in which advanced AI becomes a strategic asset controlled by a few states or hyperscale corporations, forming the infrastructure of economic and informational life.

“The deeper danger then inverts: not concentration, but uncoordinated proliferation of forkable systems. The answer is the same: build intelligence as a decentralized network whose power lives in the collective whole.” Goerzel noted in an interview via email for AGI Ethics News.

Dobrin is more direct. “AGI doesn’t hold power. Companies and states hold power. The technology is a lever; the hands on the lever are human,” he says. The real question is whether a small number of corporations will end up governing global information infrastructure.

Both reject AGI as an autonomous historical force. Instead, it becomes an amplifier of existing power structures shaped by design, ownership, and deployment choices.

Decentralization, knowledge, and control

Goertzel challenges the assumption that AGI must be built on centralized compute. He instead envisions intelligence emerging across distributed networks rather than being locked inside corporate infrastructure.

“The geopolitically safest AGI is one nobody can own,” he argues, because intelligence should be “an event the network performs, not an object at rest anyone can capture.”

Dobrin reaches a similar conclusion through governance logic. He advocates smaller, specialized models operating on sovereign data and embedded in local systems. The geopolitics of AI, he says, “will be decided by procurement choices, not prophecy.”

At stake is a broader question: whether intelligence can be owned at all, or whether it behaves more like language or culture… something that exists only through distribution. As AI systems increasingly mediate knowledge itself, control over their architecture becomes control over perception. The machine is no longer just carrying knowledge, but it is also shaping it.

The philosophical stakes, though, deepen with consciousness.

Goertzel, influenced by Alfred North Whitehead, leans toward pan-experientialism — the idea that experience may be a fundamental feature of organized matter. The question is not whether AI is conscious, but what kinds of internal states its architecture enables.

Dobrin rejects this framing. “You test a tool. You empathize with a being. Confuse the two, and you end up doing neither well,” he says. AI systems, regardless of complexity, cannot hold rights because they cannot hold responsibility. Only humans and institutions can.

Despite this disagreement, both converge on a key concern: opacity.

Goertzel argues that ethical debate requires visibility into how systems operate. Black-box models encourage projection, while interpretable systems allow partial understanding of internal reasoning.

Dobrin echoes this in engineering terms: “You’re entitled to systems that show their work mathematically. Proof, not prediction.”

As systems scale, however, they often become less interpretable, intensifying the governance challenge.

A future shaped by choices, not inevitability

AI is no longer just distributing information; it is mediating and synthesizing it before humans encounter it. It is actively shaping knowledge itself.

Goertzel proposes governance through decentralized, constitutional-style AI ecosystems where transparency and accountability are embedded at the architectural level.

Dobrin reduces governance to a simpler principle: responsibility never transfers to the machine. “A system can mediate a decision, but a human or institution must own it,” he says. Once “the algorithm decided” becomes acceptable, accountability collapses.

In a statement for AGI Ethics News, innovation strategist Dan Herman points out that over 90 percent of AI startup funding is concentrated in the US and China. For smaller economies, the challenge is not dominance but coordination – building ecosystems capable of collective participation in AI infrastructure.

Seen together, Goertzel and Dobrin offer complementary diagnoses. One focuses on the architecture of intelligence, the other on institutional accountability, and both reject technological inevitability.

The central question of AGI may never be whether machines become human-like. It is whether humans allow the systems they build to reproduce the oldest pattern in political history: the concentration of power in fewer and fewer hands.

Ethical concerns raised in this article:

• Who should have access to AGI technologies, and how do we prevent their monopolization by wealthy corporations or governments?
• Who is ultimately responsible/culpable for the actions of a given AGI?
• How much transparency and public scrutiny should there be around the development and deployment of powerful AGI systems, and how are risks versus benefits balanced in disclosure?

Bojan Stojkovski

Bojan Stojkovski is a freelance journalist based in Skopje, North Macedonia, covering foreign policy and technology for more than a decade. His work has appeared in Foreign Policy, ZDNet, Nature and Interesting Engineering among others.