In this blog post, we will examine the necessity and risks of developing strong AI and superintelligence, with the text revised and proofread to reflect current facts.
According to Ray Kurzweil, the development of artificial intelligence is expected to continue accelerating in accordance with the Law of Accelerating Returns. The Law of Accelerating Returns is a theory stating that the fundamental performance of information technology evolves along a predictable exponential trajectory. He has argued that we will eventually reach the “Singularity,” a point at which artificial intelligence surpasses human intelligence. However, the annual projections presented in the original text are merely predictions, and as of now, neither the uploading of human consciousness to a computer nor the realization of a general-purpose AI surpassing human intelligence by billions of times has been achieved. Furthermore, “Neuralink”, founded by Elon Musk, is developing brain-computer interfaces (BCIs), and while some clinical trials are underway, the technology has not yet reached the stage where human memories or consciousness can be uploaded to or downloaded from a computer.
Artificial intelligence is indeed advancing at a very rapid pace, and concerns are consistently being raised that this progress could pose a threat to humanity’s future. There are even perspectives that worry about the possibility of AI surpassing humans and taking control. So, is it really necessary to research strong artificial intelligence (AGI)—and, furthermore, superintelligence—at the risk of such dangers? Humans can sufficiently improve their quality of life even with narrow artificial intelligence capable of performing specific tasks.
Nevertheless, many companies are continuing their research and development aimed at strong AI and superintelligence—systems capable of independent thought and learning—in order to secure competitiveness and technological superiority.
Google is a prime example. In the past, Google developed AI specialized in specific fields, such as “AlphaGo,” and has since advanced various AI technologies, including generative AI and a range of AI assistants. This research is gradually expanding toward the goal of general-purpose AI. I believe that this research into strong AI and superintelligence could ultimately pose a risk to humanity.
Of course, looking at human history, one might think that humans will be able to control strong AI and superintelligence as well. Humans have, up to now, tamed and adapted to entities more powerful than themselves in various ways. In fact, humans have raised and utilized animals and plants, and historically, there have been instances of humans dominating other groups. Although this history is by no means something to be proud of from an ethical standpoint, the fact that humans have controlled various entities is difficult to deny. For this reason, many people might readily agree with the argument that humans will be able to control artificial intelligence as well.
However, among the entities humans have dominated so far, none have surpassed human intelligence. Humans have overcome their physical limitations through their superior intelligence and have been able to exert control because they secured a physical and intellectual advantage over the entities they dominated. However, if strong artificial intelligence or superintelligence—possessing intelligence comparable to or surpassing that of humans—is developed and combined with robots that exceed human physical capabilities, the situation could change entirely. This could potentially become an opponent that humanity would find difficult to handle.
Furthermore, artificial intelligence learns based on the vast amounts of data accumulated by humans. This data inevitably includes the biases, desires, and conflicts of human society. Therefore, without appropriate safeguards in place, it is difficult to rule out the possibility that these factors will influence the AI’s decision-making.
These concerns are not mere speculation. In fact, Microsoft’s chatbot “Tay” was taken offline less than a day after its release after it began spewing racist and sexist remarks influenced by users. This incident is often cited as a prime example demonstrating how AI can be significantly influenced by its training data and the user environment. It is also difficult to completely rule out the possibility that the biases or desire for domination present in human society will be learned by AI if proper controls are not in place.
However, there is also a counterargument that if certain guidelines are established before AI becomes a threat, and research and development proceed in accordance with those standards, then strong AI and superintelligence can be fully accepted. In fact, when examining the long-term issues addressed by the “Asilomar AI Principles” and the case of DNA recombination technology, one might find this argument somewhat persuasive.
Since DNA recombination technology is a field requiring a high degree of expertise, discussions in the early stages were centered on scientists rather than the general public or the community. As a result, the social and cultural aspects of the risks were not fully addressed. However, during the process of establishing guidelines by the U.S. National Institutes of Health (NIH) following the Asilomar Conference, scientists recognized the risks of their research on their own and established a self-regulatory framework, including the temporary suspension of some studies. They even set stricter research standards themselves out of concern for problems that could arise from intervention by the general public, who lack specialized knowledge. Had researchers pursued only private interests, the technology could well have been misused; however, thanks to this self-regulation, DNA recombination technology continues to be utilized primarily in the fields of medicine and life sciences, and serious social problems of the magnitude once feared have not materialized. Furthermore, the “Asilomar AI Principles” also emphasize responsible AI development by outlining research directions and the importance of ensuring safety in preparation for the era of superintelligence.
However, I believe these guidelines alone are insufficient. As mentioned earlier, what we are discussing is, after all, an entity that has the potential to surpass human capabilities. The original text compares the human brain to a level of 36.8 petaflops per second, but it is difficult to view this figure as a scientifically agreed-upon value at present. Furthermore, simply comparing human intelligence to the computational performance of supercomputers is also inappropriate. That said, it is clear that current supercomputers and AI systems possess computational capabilities far superior to those of humans in specific fields, and their computing power continues to improve.
There is also significant discussion about the possibility that entities with intelligence surpassing that of humans could learn independently and enhance their capabilities. Some believe that even if an era arrives in which serious accidents caused by AI occur, humans will be able to control them through even more advanced technology. However, in the past, humanity has driven many plant and animal species to extinction or to the brink of extinction during the processes of industrialization and technological advancement, and has also severely damaged the environment. The reality is that, having recognized these problems too late, humans are ultimately bearing the consequences once again. Similarly, the author argues that if we wait to devise countermeasures only after serious harm has been caused by the development of strong artificial intelligence and superintelligence, the result could ultimately be akin to “closing the barn door after the horse has bolted.”
We need to thoroughly discuss not only the feasibility of human control but also the economic impacts that strong AI and superintelligence will bring. Let’s assume, for the sake of argument, that humans develop strong AI and superintelligence and continue to live while maintaining control over them. The emergence of AI—that is, sentient beings—with capabilities surpassing those of humans might seem like a great boon to society. However, it remains questionable whether such sentient beings will ultimately benefit everyone.
Today, many countries around the world are facing the problem of a shrinking economically active population—those responsible for production and consumption—due to low birth rates and an aging population. This phenomenon is giving rise to new challenges, such as slowing economic growth and labor shortages, and various solutions are being discussed to address them. Some argue that robots with physical capabilities superior to those of humans and advanced artificial intelligence could replace this labor force.
However, I do not believe this will necessarily lead to the benefit of humanity as a whole. If such intelligent entities are developed, companies are likely to prefer artificial intelligence—which requires no wages, incurs low maintenance costs, and rarely gives rise to labor disputes or other conflicts. Ultimately, this could lead to job losses for a significant number of workers. A decline in labor income would likely cause consumption to shrink, and this decline in consumption, in turn, could create a vicious cycle leading to reduced corporate revenue.
Of course, many jobs have disappeared due to technological innovation in the past, but society has adapted to these changes as new industries and occupations have emerged. The economic recovery following the Great Depression of 1929 and the shifts in industrial structure after the Industrial Revolution are examples of this. However, these changes were possible in part because new roles for humans continued to exist. If general-purpose artificial intelligence that surpasses humans comes to possess superior capabilities in nearly every field, debate continues over just how many roles will remain for humans to fill.
Some predict that the economy will ultimately find a balance, as new jobs will be created to replace those lost to AI. In fact, the World Economic Forum (WEF) also analyzes that AI will replace existing jobs while simultaneously creating new ones. However, it also emphasizes that, given the expected rapid pace of change in the job market, a significant scale of career transitions and retraining will be necessary. Nevertheless, I find it difficult to believe that these changes will necessarily bring only positive outcomes for society as a whole.
Even from a social perspective, I believe a cautious approach is needed regarding strong AI and superintelligence. As technology advances, economic disparities tend to widen, and since the spread of digital technology, the digital divide has also emerged as a significant social issue. As mentioned earlier, if the human desire for control is combined with artificial intelligence, this could be a problem stemming from human intent rather than the will of the AI itself. If a specific country or power group monopolizes such technology to strengthen its military power and political influence, there is a possibility that this could pose a new threat to world peace and the international order.
Since the invention of the computer, the development of search engines and information technology has dramatically improved humanity’s ability to acquire and process information. Artificial intelligence is a technology that can further advance this ability to utilize information. However, just as computers and the internet created a new digital space, a new form of crime—cybercrime—has also emerged. Even now, various forms of cybercrime—such as personal data breaches, financial fraud, ransomware, and deepfake crimes—continue to rise. By this logic, the emergence of artificial intelligence with intelligence surpassing that of humans could create new industries while simultaneously giving rise to new crimes and security threats.
Furthermore, the question of who should be held legally responsible if such an intelligent entity commits a crime remains an unresolved issue. Today, most countries do not recognize artificial intelligence itself as a subject of criminal liability; legal judgments generally center on the responsibility of developers, operators, or users. While there is some discussion about granting artificial intelligence a legal status similar to that of a corporation, no internationally agreed-upon system has yet been established. Therefore, as the era of superintelligence becomes a reality, there is a growing need for legal and ethical frameworks to evolve alongside it.
After comprehensively examining the question of whether humans can control strong AI and superintelligence, as well as their economic and social impacts, the author argues that the development of strong AI and superintelligence—which are on par with or surpass human capabilities—is an issue that must be approached with the utmost caution. Current AI is advancing rapidly based on deep learning, machine learning, and large-scale data analysis technologies, and research into brain-computer interface technology—which connects humans and computers—is also steadily progressing. If these technologies are utilized to complement human capabilities, we can expect greater synergistic effects, and they can make a positive contribution to the development of human society.
In other words, the author argues that humanity’s progress does not necessarily depend on developing superintelligence that completely replaces or surpasses humans. Society can reap sufficient benefits even from AI that collaborates with humans. In fact, various forms of weak AI—such as generative AI, medical AI, autonomous driving assistance systems, and smart home appliances—are already significantly transforming our lives, and their scope of application is expected to expand even further in the future.
Ultimately, what matters is not reckless technological competition, but rather a social consensus on how far to advance AI and how to utilize it. Rather than attempting to halt technological progress itself, it is far more important to develop and utilize AI in a way that prioritizes human safety and dignity above all else. I believe we have reached a point where we need the wisdom to advance AI to a level that contributes to the sustainable development of human civilization, rather than repeating the mistake of “going too far” through excessive competition for advancement.
