Militarization of Artificial Intelligence and Its Security, Ethics and Legal Challenges

By Zhu Qichao

Director and Professor, Center for National Security and Strategic Studies, College of Advanced Interdisciplinary Studies, National University of Defense Technology & Research Fellow with the Think Tank for Defense Technology and Strategic Studies

In recent years, with the rapid growth of big data, the remarkable improvement of computing power and the progress of deep learning algorithm, artificial intelligence (AI)has built the third wave of development after two ups and downs, which is deeply impacting the political, economic, military, social, cultural and other fields of various countries.Various countries in the world, especially the major military powers, are accelerating the military application of artificial intelligence.Thus, the nature and form of war are undergoing profound changes.As a new subversive technology, what kind of prospect will AI bring to the future war? How will a new round of military change be promoted? What challenges and worries will the AI military application bring? This paper tries to make a preliminary exploration on these issues.

I.The historic evolution of warfare driven by artificial intelligence

War is a special activity of human society.It refers to the organized and planned use of force to carry out intense military confrontation for certain political and economic purposes.It is the highest form of struggle to solve contradictions and conflicts among classes,nationalities, political groups and countries.It is the continuation of politics through violent means.Carl Clausewitz pointed out that war is the continuation of political interaction through another means.The form of War refers to manifestation of the external form of war or external manifestation of the inevitable connection of its internal essence.It is determined by the economic situation, political nature and military development level of the period.The change of technological form influences the change of military form and promotes the transformation of war form.In other words, the breakthrough and application of the symbolic (subversive)technology of an era promotes the fundamental change of the structure and function of military system, i.e.,qualitative change in the main carrier of operation, tools of operation, space of operation,mode of operation, establishment of military system, military theory, etc.Only in this way can the qualitative leap of military system be promoted and the form of war be transformed accordingly.In this way, the basic elements of the form of war include the main body of war,the tools of war (weapons and equipment), the space of war (battlefield), the style of war (war fighting), the theory of war, the configuration of armed forces and other specific aspects.The transformation of the form of war means that some or all of these elements have undergone significant changes, which promotes the war to jump to another form.

Throughout history, major technological breakthroughs and applications have often brought revolutionary changes to war.Since ancient times, there are roughly four major military revolutions in human society.The first military revolution is a metallized military revolution in which metal weapons such as bronze and iron replaced the stone aged weapons such as stone, sticks weapons; the second military revolution is a gunpowder military revolution in which hot weapons replaced cold weapons; the third military revolution is a mechanized military revolution brought about by the emergence and popularization of mechanized equipments; and the fourth military revolution is a informationized military revolution in which information technology innovation led the advancement of information-based weapons and equipments into the battlefield.Between the third and fourth revolutions, there is also a military revolution marked by nuclear weapons.It is easy to see that the triggering of these four military revolutions is inseparable from the emergence and development of disruptive technology in that era.From smelting technology to propellant technology,mechanization technology, atomic energy technology and information technology, the four military revolutions are affected by the core role of technological revolution.In line with the four military revolutions, human society has gone through four major forms of warfare: cold weapon warfare, hot weapon warfare, mechanized warfare and informationized warfare.From the appearance of the formal war of mankind to the beginning of the 19th century, it belonged to the era of cold weapons war.During this period, the dominant weapons and equipments were cold weapons such as knives, axes, rods, spears,bows, crossbows and others.The main way of releasing energy from weapons and equipment was to transform physical energy into mechanical energy, which mainly tested the physical ability of combatants.In the era of thermal weapons war, the dominant weapon equipments are gun and artillery based on gunpowder.The main way of weapon destruction is to convert thermal energy into chemical energy, which mainly tests the skills of combatants.The era of mechanized warfare is roughly from the beginning of the 20th century to the middle of the 20th century.During this period, aircraft, tanks and other mechanized weapons and equipment platforms began to mount the stage of warfare.The main way of energy release is the conversion of chemical energy into mechanical energy.Since the middle of the 20th century, informationized weapons such as precision-guided weapons have stepped onto the stage of history, and human society has gradually entered the stage of information warfare.

Currently, mankind is still in the era of information warfare.Meantime, with the rapid development of artificial intelligence (AI)technology, and the emergence of concepts and weapons related to military AI, countries are actively adjusting the existing military theory and institutional configuration, and the trend of the transformation of warfare to military AI is becoming more and more obvious.The main carrier of war will change from the original military organization, state and national alliance composed of humans to the military organization, non-military organization,non-state organization and robotic force composed of humans; the form of military revolution will change from metallization,gunpowder, mechanization to informationization, intelligence and unmannedness; the dominant factors of war game will transform from the original material and energy to information and intelligence;warfare style will change from cold weapon killing to robotic warfare, cyber-centric warfare,algorithm warfare, cognitive warfare, etc., with the characteristics of hybrid warfare becoming richer; confrontation mode will change from weapon confrontation and platform confrontation to system confrontation and man-machine combined confrontation;battlefield space will further expand from physical space such as land, sea and air to space, cyber, electromagnetism, biology,cognition and other fields, with cross-domain operations becoming normal.The branches of traditional forces will continuously be reinforced by the new branches such as cyber forces, space forces and robotic forces.

II.Artificial Intelligence Promotes a New Round of Military Reform

It is generally believed that when new military technologies, weapons and equipments,operational concepts and organizational configurations interact to significantly enhance military operational capabilities, new military changes will be promoted.Artificial intelligence is more and more widely used in the military field, is becoming important pusher of military reform, has promoted the emergence of new war patterns and changed the internal mechanism of victory in war.On the one hand,the AI employment will probably shape subversive military capabilities, which will lead to a doubling or a substantial leap of combat effectiveness.On the other hand, it will also bring new challenges to the innovation of military theory and the practice of military capabilities.

(1)Impact on the traditional concept of war

The history of human warfare has gone through the cold, hot, mechanized and information age.Artificial intelligence has accelerated the arrival of the era of intelligence.For military capacity building, if mechanization is the foundation and informationization is the nerve and blood, intelligence embodies the ultimate confrontation between human intelligence.Can intelligence be divided into high-tier intelligence and low-tier intelligence?Does the army with high-level intelligence have an overwhelming advantage over the army with low-level intelligence? If combat effectiveness is the man, weapon and combination of man and weapon, then, how to define the relationship between man, robot and intelligent information system in the era of intelligent warfare? If the "mechanization" of human beings and the "humanization" of machines are two inevitable trends of development, would it be contrary to the traditional war ethics that thinkable robots replace human beings to fight? Artificial intelligence makes the battlefield perception and information processing capabilities unprecedentedly improved.Does the "dense fog" of war still exist in the high-tech battlefield? Understanding these issues requires a brainstorming concept renewal in the military field.

(2)Promoting weapons and equipments and command decision-making intelligence

On the one hand, AI as an enabling technology will be embedded in the traditional weapon system to enhance its intelligence and combat effectiveness.On the other hand, AI will also give birth to new weapons and equipments, such as highly autonomous unmanned aerial vehicles, unmanned vehicles,unmanned submarines and other new unmanned combat platforms, and "unmanned bee colony" and bionic robots.In the field of intelligent command and control, military forces of various countries develop diversified military information systems in order to build a powerful grid of cyber information system and improve the ability of intelligent evaluation and auxiliary decision-making.In recent years, the U.S.military has established and upgraded the Cyber Command, and vigorously strengthened the building of cyber attack and defense capabilities, focus on intelligent diagnostic information system for cyber intrusion based on the research and development of cloud computing, big data analysis and other technologies, which can automatically diagnose the source of cyber intrusion, the degree of attacker’s cyber damage and data recovery capability.In addition, the U.S.military is promoting the employment of artificial intelligence in defense intelligence analysis,assisting the DoD and other military levels in war decision making, and improving the scientific, effective and reliable nature of intelligence analysis and decision-making.On April 26, 2017, Robert Walker, deputy secretary of defense, signed the memorandum and authorized the establishment of the"algorithm war cross functional group", aiming to "rapidly transform the massive data of the DoD into usable information, effectively promoting the application of artificial intelligence, big data, machine learning and other technologies in the field of military intelligence".It is not difficult to predict that AI will gradually enter the key fields of war design,strategic policy and operational command, etc..The human-robot integration mode will become the basic form of command and decision-making in the future.

(3)Promoting the innovation of military theory

The constant updating of the material and technological basis of war and the continuous evolution of the form and style of war naturally open up a new space for the innovation of strategic theory and concept of war.For example, with the vigorous development of information technology, its role of its catalyst,adhesives and efficiency multiplier has become increasingly prominent, which has led to the continuous emergence of new disruptive technologies in the field of artificial intelligence.On the other hand, it has also made it possible for those who are in the technological advantage to provide "strategic guidance" and "technical deception" to imitate followers; the combined application of precision ammunition, unmanned equipments and cyber information system has led to the emergence of new intelligent warfare theories such as "distributed killing", "mother ship theory", "battle cloud" and "bee colony tactics".Depending on the information superiority and decision-making superiority of one's own, how to cut off and delay the opponent's information and decision-making loop in the decentralized battlefield network has become the core problem to be solved in order to win an intelligent war.It can be predicted that"unmanned cluster warfare" and "man-machine cooperation warfare" may become the mainstream warfare mode in the future.The distributed operational unmanned trunking system has the capability of "observation and combat integration", such as reconnaissance and surveillance and autonomous attack, has high cost performance and can be recovered.It has great advantages in saturation attacking on the enemy.The U.S.military is stepping up the research of "unmanned bee colony" represented by "elves" and other projects to verify and evaluate the feasibility of low-cost unmanned systems clustering technology.In addition,man-machine cooperation operations are increasingly favored by various countries'armed forces, and the "warrior" explosive ordnance robot which cooperates with soldiers has demonstrated its strength in the battlefields of Afghanistan and Iraq.

(4)Renewing the means of military education and training

The AI wide application in the military field will lead to new changes in the military education and training.Weapons, equipments and systems that win in battlefields are becoming more and more intelligent, and are more and more closely connected with scientists, advanced technology laboratories and battle laboratories.So the situation that scientists design war, military strategists command war, and artificial intelligence wins war will come.Traditional education and training subjects will be further compressed.Intelligent training platform in the form game-playing, experiential virtual reality technology and augmented reality technology will provide new means for military training.Knowledge exploring and data exploring process supported by super-large-scale computing, cloud computing and big data technology are themselves important parts of ability building.The development of AI technology will also provide new means for the evaluation of military personnel's physical skills, physiological functions, psychological effects and other levels of education and training.On June 18, 2018, the national defense authorization act for 2019 fiscal year, which was formally passed by the U.S.Congress,specially set up a section entitled "bringing advanced technology into vocational military education", which stressed the need to strengthen the application of advanced technologies such as artificial intelligence in vocational military education, and requested the Secretary of Defense to report to Congress before December 1, 2018 the feasible plan of embedding AI technology into vocational military education projects, concretely including the establishment of appropriate vocational military educational institutions, and expansion of enrollment by vocational military education institutions in private sectors.

III.Security, ethics and legal concerns of AI militarization

The AI is accelerating its penetration into the military field, which not only brings tremendous opportunities for military change,but also contains many risks, generally speaking, in the following aspects.

An AI arms race and strategic instability."Strategic stability" means that when potential opponents realize that conflict with each other is difficult to gain, the two sides will not act rashly, resulting in strategic stability.According to the theory of Offensive-Defensive balance, the strategic stability will be strengthened when the defensive is dominant, while the strategic stability will be weakened when the offensive is dominant.The emergence and employment of lethal AI weapons will have a great negative impact on strategic stability and threaten regional and international security.Specifically,AI weapons may enhance the dominance of offensive side and have a series of serious consequences, including preemptive strike to be the best means to gain strategic advantages,the best defensive strategy being to launch preemptive strike, which will lower the threshold of the use of force among countries and increase the possibility of an arms race.In addition, the use of AI weapons will pose a huge problem to the accountability of users,which will easily lead to strategic misjudgment and destroy strategic stability.Furthermore, the asymmetric nature of armed conflicts between countries will be magnified by the AI weapons going to the battlefield.The gap between countries with advanced technology and the ability to develop, procure and deploy AI weapons and those without these capabilities is likely to continue to magnify the asymmetric nature of armed conflicts in the future.Besides, the strategic stability of major powers is largely based on the nuclear deterrence of"mutually assured destruction", i.e.both sides have the capability to deter the counterpart's pre-emptive second nuclear strike.However,the rapid development of sensor technology at present may make the retaliatory weapons such as submarines and mobile missile systems easier to be detected, located and destroyed,and increase the vulnerability of the secondary strike ability, thus eroding foundation of the nuclear deterrence system.According to reports,the US DoD is funding a mobile missile launch platform aimed at using artificial intelligence to help identify and predict nuclear missiles launche, as well as tracking and targeting North Korea and other countries.If true, it will undoubtedly greatly undermine strategic stability.According to a report by RAND Corporation, AI may trigger a large nuclear war before 2040.

The risks of proliferation and malicious use of AI weapons and equipments.AI is a neutral technology, but users are not neutral, so there is a risk of malicious use.This wave of AI mostly appears in the private sector.AI technology itself has obvious dual-use and proliferation, and AI software can be replicated and disseminated at almost zero cost, which also reduces the threshold and difficulty for users to acquire such technology.As long as one has enough funds, it can effectively improve the cyber attack ability without requiring too high technical capability.The advantages of AI technology, weapons and equipments have great attraction for non-state actors such as terrorist organizations and transnational criminal groups, can multiply the strength of force, effectively reduce the cost of attack, save their own combatants and make them more capable of launching terrorist attacks.For example, there are already technologies for developing unmanned aerial vehicles to transport explosives.Once such technologies fall into the hands of terrorists,they will provide new means and methods for the single wolf terrorist attacks, making it more difficult to prevent and trace accountability.Therefore, the development of AI weapons may aggravate the threat of terrorism and worsen global or regional instability.In addition,research shows that AI technology can be used not only to collect and analyze data, but also to produce false information including automatically generated images, videos and texts.It is not difficult to predict that the AI capacity to produce false information such as"false news" and "false intelligence" will be used maliciously, which will surely cast another layer of "dense fog" on the war, and make it more difficult to distinguish the true and false information in the future battlefield.

AI limitations.Although AI technology has many advantages, it also has many limitations, which can be divided into internal and external limitations.There are three main internal limitations: one is "narrowness".The current AI technology basically belongs to the category of "weak AI" or "narrow AI".Although it can show great superiority over human beings in systems with definite rules and complete information, it still needs human intelligence such as rational analysis ability,flexible response ability and moral identifying ability when solving war problems beyond programming.In this respect, AI is unable to replace and surpass human intelligence.In other words, current AI technologies and systems lack the ability to understand generalized context, lack human knowledge and flexibility, which is also a "Moravec’s Paradox".The second is "vulnerability".This means that once the real world environment changes, AI systems may fail or even disrupt,from "super smart" to "super stupid".If such AI systems are used in highly changing military battlefields, once confusion occurs and operators fail to stop it timely, catastrophic consequences may be generated.For example,the launched missiles by the automatic missile early warning system with false warning may lead to accidental injury or even escalation of the crisis.Third, it is difficult to explain and predict.Unlike traditional information technology, AI technology has high complexity, and may easily bring about uncertainty and unpredictability.Concretely, AI has inherent defects of "black-box algorithm".Its mechanism is difficult to be fully understood by human beings who involve in,deploy and operate such machines, and the results are more difficult to be predicted and guaranteed.In addition, when different schemes and configurations of systems or the interaction between systems and codes is too fast, such risks may be aggravated, and system failures caused by system self-interaction are difficult to eliminate.On the other hand, the external limitations mainly come from human-computer interaction errors, cyber network attacks (such as encountering disastrous consequences of enemy hacker attacks, virus data injection, etc.).AI weapons and equipments need to rely on computer software and systems to operate, and software and systems are easily attacked by enemy network hackers, resulting in the risk of losing or even changing side on the battlefield.

(2)Ethical Concerns

In 1942, the famous American-Russian writer, Asimov in his the "Amazing Science Fiction" first proposed the ethical problem of robots, and set the "three laws for robots" in order to constrain the behavior of machines and make them comply with the mandatory ethics of protecting human beings.Nowadays, the AI rapid development increasingly upgrades the autonomy of robots and gradually moves to battlefield, which also poses a significant challenge to traditional war ethics.

The increasingly autonomous weapons and equipments will impact the traditional man-machine relationship.The relationship between man and technology-supported weapons and equipments constantly evolves in history.For a long time since the emergence of human social warfare, no matter how weapons and equipments get developed, they after all are only tools in the hands of human beings, but only differ in lethality, complexity and precision.In other words, during this period,human beings are always the absolute controllers of weapons and equipments.Weapons and equipments are only the extension and expansion of human physical strength and intelligence, and do not pose a direct challenge to human beings as a whole.Nowadays, the rapid development of AI technology may make weapons equipments more and more autonomous.In the long run, if war robots develop self-consciousness and have the judgment and decision-making ability similar to or even beyond human beings, and changing from technical substitution to decision-making substitution, will pose a subjective threat to the survival of mankind as a whole, seriously impacting the traditional ethics and morality of human beings and human dignity.In the near future, if it is technically feasible, should the"decision-making power of life and death" be transferred to inanimate machines so that they can launch attacks independently on the battlefield to determine human life and death?This problem will pose a serious challenge to the Asimov Law of "no harm to human".Should human ethics be implanted into increasingly intelligent machines, what kind of ethical norms should be implanted and how to implant them? Should the development of lethal autonomous weapons be restrained?These problems will become the difficult matters that AI has to face when it gets to military employment.

Intelligent weaponry challenges traditional war ethics.With the trend of AI weapons moving towards battlefield, the traditional war ethics such as the justice of launching a war, the justice of fighting a war,and the justice of ending a war will be seriously challenged.Concretely, in terms of war justice,more and more non-state actors can also acquire cheap and easy-to-spread AI weapons and equipments.Once they launch an attack on state actor, the elements of legitimate authority and legitimate reasons will be impacted.In addition, with more and more AI-supported unmanned weapons and equipments going to battlefield, the casualties concerns of the national war options are weakened, and the threshold of war may be significantly lowered,and small-scale wars dominated by unmanned combat platforms are more likely to break out.In terms of warfare justice, it is difficult for the current AI weapons to distinguish between military troops and civilians, military facilities and civilian facilities.Once put into use in a battlefield, it may cause large-scale accidental killings and injuries, resulting in innocent civilian casualties.What's more, if the lethal autonomous weapons without human supervision going to battlefield, because of its high degree of autonomy, lethality and emotionless, they will have no sense of guilt and compassion for killing living beings, which will seriously impact the bottom line of human ethics and morality.In the aspect of ending-war justice, the military application of AI weapons and equipments may cause the predicament to decide the subject of war responsibility.In the past wars, only human beings are the only legitimate subjects and responsibility bearers in wars.But with the increasing autonomy of battlefield robots and intelligent systems, can robots become moral subjects and responsibility subjects and shoulder unjust and illegitimate responsibilities in war operations?This problem will become increasingly highlighted.

(3)Legal Concerns

In addition to security and ethical concerns, the AI militarization has also brought many legal problems.

War accountability.The development and employment of AI weapons and equipments may lead to a "blank accountability" problem, the main reason of which is that the attribution of responsibility will become more and more difficult.For the misconduct caused by AI weapon systems,especially UAVs operations, the accountability investigation and allocation may involve many levels such as state actors, individuals and weapon systems, covering the whole process of system R&D, producing, arming and employing.Therefore, the event-related operator, manufacturer, programmer, purchaser,arms dealer, military commander, equipment support personnel and even the weapon system itself are to blame, and the difficulty of tracing and distributing responsibility will be greatly enhanced.How to allocate relevant responsibilities? Should we trace the legal liability of robot crime? How can weapon systems feel punished and deterred without life and self-awareness? What is the legal and practical value of tracing its responsibilities?These problems remain unresolved.In the future, the two trends of "mechanized man"and "humanized machinery" will become more and more obvious, by then, the allocation of war responsibilities may face greater difficulties.

The core principles of international humanitarian law will face profound challenges.Under the background that AI is moving towards battlefield increasingly, the core principles of international armed conflict law, such as the principle of necessity, the principle of distinction, the principle of proportionality and the principle of humanity,will be challenged severely, facing the problem of how to apply and adjust them.Concretely,the principle of necessity means that the necessary operational actions taken in order to make the enemy yield are justified, but obviously unnecessary military actions are prohibited.The principle of proportionality means that when attacking military targets,incidental or collateral damage to civilians and civilian objects should be minimized.The collateral damages caused in military operations should not exceed the damages produced in order to achieve the expected direct and specific military advantages.The principle of distinction refers to the distinction between combatants and civilians and the protection of victims in armed conflicts.The principle of humanity means that all human beings involved should be treated humanely.Currently, robots in battlefield are still unable to effectively distinguish between combatants and civilians, and indiscriminate killing of innocents will pose a major challenge to the principles of distinction and humanity once used in battlefield.The collateral damage caused by the use of robots in battlefield will challenge the principles of proportionality and humanity.The overflow of UAVs impact the necessary principles by lowering the threshold of war.Currently international law is undoubtedly human-centered, but can robots become legal subject? Is the existing law of international armed conflict sufficient to regulate the increasingly autonomous AI weapons at present and in the future? Is there a need to amend or enact new international humanitarian law? How to make it? These matters need further study and discussion.

An objective view of the security, ethical and legal challenges that may arise from the AI militarization is the premise for us to understand AI, develop AI, manage and control AI.The characteristics of AI itself have brought difficulties and even obstacles to people's understanding of AI.An objective view of the security, ethical and legal challenges that may be generated by the AI militarization is first of all necessary to establish correct conception.

Firstly, upholding a scientific attitude.The security challenges brought by AI may come from immature technology, or from the negligence of human management and control,so it is necessary to conduct a practical and realistic analysis with a scientific attitude.Generally speaking, the security problems caused by AI can be divided into three levels:firstly, the security risks in the management of proper operation rules within an industrial sector, an aspect or a system; secondly, the security challenges at the national security level to the fields of politics, economy, finance,science and technology, national defense;thirdly, challenges caused by the transnational use of AI technology such as the cyber security,space security, nuclear security,counter-terrorism operations and even international security governance.The security challenges brought by AI at different levels have different characteristics and should be responded accordingly, which need to be treated differently rather than put in one basket.

Secondly, adhering to the attitude of development.In overview, the security challenges brought by AI are still in a controllable range.People are worried about the future development of AI mainly because AI as a technology may trigger more security problems.Historically, at the beginning of each industrial revolution or technological revolution, people are panicked to a certain extent because of the relative lack of knowledge and practical experience of the technology.For example, in the early days of the first industrial revolution, workshop workers boycotted and destroyed machines,and in the early days of the second industrial revolution, people panicked about the casualties caused by improper use of electricity.As a product of the evolving human civilization,we need to look at the positive role of AI and its security challenges from the perspective of development, and should bring the potential of AI into full play for the welfare of mankind,but also manage and control the security crisis that AI may cause in advance.The more people can objectively view the security challenges brought by AI, the more they can explore the blue sea of AI security technology and security industrial innovation.

Thirdly, sticking to the attitude of cooperation.AI technology is closely related to the development of information technology,especially technological breakthroughs in mobile broadband communication, big data,cloud computing, Internet of logistics and other fields will promote the maturity of AI-related technologies and industries.From the technical perspective, AI technology is similar to information technology, which has the characteristics of low cost, easy diffusion and difficult control, but once this technology matures, the marginal cost of the market will be greatly reduced, even considered as a technology of "zero marginal cost".This makes the security problems that may be caused by AI easy to spread and difficult to control.For example, if there is a failure of supervision, a hacker may use new AI algorithms and intrusion technique to disrupt normal operation of a country's power grid, public transport control network and so on, causing large-scale catastrophic consequences, and even harming innocent people.In theory, the higher the level of information society and intelligence, the more the dependence on the technology system will be.Once the original technology system is impacted, new uncertainties and vulnerabilities will arise.In the rapid development era of information technology and AI technology, the security challenges we are facing are characterized by technical complexity and system vulnerability.Because of this, we must adhere to the attitude of openness and cooperation.Government departments, social sectors and individual citizens should together strengthen the technological cognition and practice experience of AI, and jointly respond to the security challenges which artificial intelligence may bring, those who would act alone will get more and more delinked with the era.

In short, regarding the wave of AI militarization, we can not just see its benefits to blindly praise it, nor hesitate to move forward because of its potential security, ethical and legal challenges.We should objectively understand the development and application of AI.If misunderstanding, misjudgment and misuse of AI-related security matters can be constantly avoided in future practice, human kind can still command its own destiny.