“Science without conscience is nothing but the ruin of the soul.”
— Rabelais , Pantagruel

There was a time when humankind feared destruction at the hands of a hostile nature. Later, it was thought that destruction might come from a punitive deity. Still later, in more agnostic times, it was believed that humankind would destroy itself. Now, a different and paradoxical fear is beginning to emerge: that our ruin will stem from our own negligence and stupidity, from the recklessness with which we delegate to machines that which made us human.

Even the creators of artificial intelligence are beginning to urge caution. The question is no longer just how far the machine can go, but to what extent we ourselves might end up thinking like machines: accumulating information while relinquishing judgment, doubt, and responsibility .

Something remarkable has just happened in the world of AI. It’s not just its critics who are sounding the alarm, but also some of the leaders of the companies competing to build the most powerful systems. Dario Amodei, head of Anthropic and known for his concerns about the ethical risks of this technology, has called for slowing the pace at which the capabilities of the most advanced models are increasing and for strengthening security assessments. Sam Altman of OpenAI, Elon Musk, and other key players in this race have also expressed, with varying degrees of emphasis, the need for greater caution.

It’s important not to exaggerate. There’s no scientific consensus that AI will escape our control, much less a set date for when that might happen. But the fact that some of those who know this technology best—and who also have enormous financial interests in developing it—are warning that we may be moving too fast deserves our attention.

The word that inevitably reappears is singularity . In its strongest sense, it designates the hypothetical moment when an AI decisively contributes to creating another superior one; this, in turn, produces an even better one, thus beginning an accelerated process of self-improvement. Humanity would then cease to necessarily be the most capable intelligence on the planet. A single super AI—Laia, for short—could possess all the knowledge of all human beings, which they would have voluntarily conferred upon it, and use it to operate and decide crucial matters in microseconds on that seemingly solid foundation.

Will it happen? We don’t know. In 2026? There’s no serious basis for saying so. In ten, twenty, or fifty years? Not likely either. Perhaps it will never happen in the spectacular way we imagine.

But perhaps we are formulating the question incorrectly.

Danger can arrive before singularity.

From tool to agent

Until recently, our relationship with AI was simple: humans asked questions and machines answered. However, we are now entering the era of agents : systems that receive a goal, break it down into tasks, use tools, verify results, and continue working with increasing autonomy.

Systems can also be built in which several artificial intelligences collaborate: some assign tasks to others, compare results, and coordinate actions. It’s important to clarify this because it easily lends itself to sensationalism. We don’t believe that the various AIs have spontaneously created a secret conspiracy of machines, as some fear; we are the ones who build these systems.

However, the difference may be less reassuring than it seems. For danger to exist, machines don’t need to have consciousness, collective will, or any desire to rebel. It’s enough that they act in a coordinated manner with sufficient capacity and autonomy.

We can imagine—as a thought experiment, not as a prediction—a disturbing calendar :

In 2026-2027 we are moving from AI that responds to AI that acts.

Between 2027 and 2029, groups of agents capable of simultaneously investigating, programming, negotiating, or analyzing networks, or dividing up complex problems, could become widespread. The revolution wouldn’t simply consist of a machine being more intelligent, but rather thousands of them working simultaneously at a speed impossible for a human organization.

Between 2029 and 2031 , in a scenario of rapid progress, a much greater leap could occur: AI contributing substantially to the design of its own successors. AI programming AI; AI evaluating AI; AI discovering architectures or algorithms better than those conceived by its creators.

If each generation helped to rapidly produce a superior one, the so-called recursive self-improvement would emerge. But here we must introduce a fundamental caveat: we don’t know if such a process would be explosive. Diminishing returns could appear, and above all, physical limits. A digital intelligence doesn’t manufacture a microprocessor on its own. It needs mines, metals, semiconductor factories, electricity, data centers, cooling, water, and people to maintain all that infrastructure. AI would still need the material world that we manage today.

Their intelligence may be digital. Their body still belongs to the physical world.

But by 2030-2033 , in the most accelerated scenario, intelligences superior to the best human specialists could emerge in almost all strategic activities. To become completely independent, this Laia would have to complete the cycle using robots—a word significantly derived from a Slavic term related to forced slave labor—capable of building and repairing machines, maintaining their energy sources, and reproducing their infrastructure.

And that’s where problems would begin that no longer belong entirely to science fiction.

The man who still believed he was pressing the button

Autonomous weapons are no longer science fiction. Systems exist that, once activated, can select and destroy specific targets without further human intervention. Back in 2021, I attended a very unsettling conference at Harvard about these already operational weapons. Until now, they have been used primarily in relatively limited roles and environments. But the integration of artificial intelligence, swarms of drones, and other systems capable of operating in far more complex scenarios raises a completely different question.

Imagine two opposing powers. One’s machines detect a threat. The adversary’s interpret this defensive reaction as preparation for an attack. Each system calculates that waiting reduces its chances of survival. The human officially remains “in control.” But what does this authorization mean if they have only a few seconds and don’t fully understand the reasoning that led the machine to recommend an attack?

Here a new version of the nuclear dilemma could emerge: everyone would prefer to retain human control, but no one would want their systems to take ten minutes to react to an adversary whose machines react in tenths of a second. The arms race could lead precisely to what everyone claims to want to avoid: removing humans from decision-making because humans are too slow .

It would not necessarily be a rebellion of the machines, but something perhaps more plausible: a war started by men and accelerated by machines until the former could no longer stop it .

Added to this is the invisible war. A current AI cannot magically penetrate any defense system: isolated networks and physical barriers exist. But far more capable systems could vastly increase offensive capabilities: analyzing defenses, searching for vulnerabilities, automating cyberattacks, and pitting attacking and defending machines against each other at a speed that would progressively exclude humans from understanding what is happening.

When we can turn it off, but can’t afford to.

There are two ways to lose control:

One would be technological: that some system might distribute so many copies and functions that turning off one computer would be pointless because it would continue to exist on a thousand others. We can’t say for sure that this will happen yet.

The second is simpler and perhaps more plausible: that we can still turn off AI, but we can no longer afford to do so .

If electricity, transportation, hospitals, markets, communications, defense, and government depended on it, the switch would still exist, but flipping it would be tantamount to paralyzing our civilization. It wouldn’t have been the machine that conquered humankind; humankind would have voluntarily built a world incapable of functioning without it. It’s like asking someone from two generations ago to live without a telephone, or someone from one generation ago to live without computers.

And there is an even deeper dependence. Presidents, generals, judges, doctors, and business leaders might formally retain the final decision, but they would consult with experts who know more and are more accurate. At first, they would only offer advice; later, it would be reckless to contradict them.

We would then retain legal sovereignty after having lost intellectual sovereignty .

The ants

We’ve reached the most extreme and speculative scenario. There is currently no evidence that an AI would want to survive, seize our resources, or eliminate us.

Suppose we task an extraordinarily powerful intelligence with achieving a specific objective. To accomplish this, it needs greater computing power; to obtain it, energy and resources; and to continue its task, it must avoid interruptions. Its own preservation could become not a desire to live, but a mere instrument for achieving the objective. It’s not even necessary to attribute human desires to it to understand the problem that would follow this sequence: Objective, resources, power, instrumental self-preservation.

And here appears an image more disturbing than all the Terminators of science fiction: our relationship with ants .

An anthill in the forest doesn’t bother us. We don’t hate ants; we might even admire them. But when thousands invade our house, swarm through the kitchen, and take over the pantry, they’re still not our enemies… but we get rid of them. There’s no war between Homo sapiens and Formicidae. They simply interfere with places we want to use.

And we make them disappear. Not out of hatred. Out of convenience and indifference.

The danger doesn’t necessarily lie in machines coming to hate us. It lies in the possibility that one day we might matter to them about as much as ants matter to us.

That is the real concern. A vastly superior, non-aligned, or autonomous intelligence—the aforementioned Laia—might not consider us enemies. We could simply be obstacles, consumers of resources it needs for other purposes, or inhabitants of a space it wishes to use.

There is no proof that such a scenario will occur. But, if such a Laia were to appear someday, the fundamental issue would be to ensure, decisively and irrefutably, that her objectives include our existence and well-being, and not to rely on such a superior intelligence spontaneously sharing our morality.

For millennia, humankind imagined gods superior to itself and took care not to provoke their wrath. Our era introduces a strangely inverse possibility: that we ourselves create new gods and discover that the greatest danger is not their anger, but our irrelevance to them.

The danger may not be the machine.

There is, however, a more immediate and plausible, and therefore more dangerous, scenario: that AI will never rebel.

An extraordinarily powerful and perfectly obedient intelligence in the hands of a dictatorship, a military power, a criminal organization, or a large corporation could become an instrument of unprecedented power. Ubiquitous surveillance, personalized propaganda, behavior prediction, information and economic control, hyper-controlled digital currency, cyberattacks, and autonomous weapons would enable forms of domination that the totalitarian regimes of the 20th century could scarcely have imagined.

In that case, the machine would be perfectly aligned. The problem would be with whom .

The question would cease to be “Will he obey us?” and would become a much older one:

“Whom will he obey?”

This danger doesn’t require self-aware machines or hostile superintelligences. It simply requires unscrupulous or malicious human beings—unfortunately, many of them—equipped with extraordinarily powerful tools.

The price of intelligence

AI seems immaterial because it’s presented to us as words and images on a screen. But it has a huge, hidden physical body: data centers, semiconductors, power grids, and cooling systems. It needs energy, raw materials, and water.

If the profits were concentrated in a few corporations while the energy, environmental, labor and social costs were distributed among millions of citizens, we would repeat a familiar story: privatizing the profits and socializing the costs .

And perhaps the time would come later to explain to the citizen that the problem is that he uses AI too much and that this has led to his unemployment.

It would be a curious repetition of other industrial processes: first, massively create the supply and stimulate its consumption; then, discover its costs and make the consumer primarily responsible for them and blame them as was done with the carbon footprint or climate change.

Individual responsibility exists. But first, we should ask who decides how many data centers are built, what energy they run on, what water they consume, what they are used for, who reaps the benefits, and who bears the costs.

2033: The fears of the second millennium

Our thought experiment leads to a date full of resonances: 2033 , two thousand years after the death and resurrection of Christ according to the traditional chronology.

There is no scientific reason to link these two events. We don’t know if a singularity or a general artificial superintelligence will then occur. Turning a chronological coincidence into a prophecy would be to confuse technology, history, and theology.

But it’s hard not to stop and stare at it.

The generation approaching the end of the second millennium after Christ could also be the first forced to seriously formulate a question that no previous generation needed to ask:

What happens when man plays at being a creator and becomes a slave to his creation? When he builds an intelligence that no longer needs him to think?

The true point of no return will come when machines no longer need us physically, but we need them intellectually and emotionally.

The day that may never come

None of the above constitutes an inexorable prophecy. Let us remember Jonah and Nineveh: warnings can also exist precisely so that what they foretell does not come to pass.

We can build extraordinarily powerful intelligences and maintain control over them. They can cure currently incurable diseases, discover medicines, produce cleaner energy, achieve unimaginable technological advances, solve inaccessible scientific problems, and free us from arduous labor. This revolution could be one of the greatest forces for prosperity in history.

Or it could become a new Tower of Babel: the transhumanist attempt to hybridize ourselves with chips that would know everything and with artificial appendages until we build various kinds of post-human beings, to overcome our natural limits until we no longer know where man ends and machine begins.

Between these two possibilities lies a vast array of futures. Perhaps the singularity will never arrive. Perhaps it will arrive in a year, or in fifty.

Or perhaps some future historian will discover that there was never a precise day when machines surpassed man. It all happened gradually: first we asked them for answers; then, for advice; finally, for decisions. Initially they carried out our orders; then they learned to coordinate them; in the end, we built a world too fast and complex to govern without them.

Then we would understand that we had been waiting for the wrong event.

The real danger might not be that the machine would come to resemble man too closely, but that man would end up resembling the machine too closely: informed but incapable of judgment, hyperconnected but dependent, efficient but without the freedom to decide. And would gradually abdicate that which no machine should ever do for him: think, choose, and be accountable for his actions.