Allen, C., Varner, G. & Zinser, J. (2000). Prolegomena to any future artificial moral agent. Journal of Experimental & Theoretical Artificial Intelligence, 12, 251 – 261.

author = {Colin Allen and Gary Varner and Jason Zinser},
title = {Prolegomena to any future artificial moral agent},
journal = {Journal of Experimental & Theoretical Artificial Intelligence },
year = {2000},
volume = {12},
pages = {251--261},
}

Author of the summary: Corrie Bouskill, 2012, corriebouskill@gmail.com

The actual paper can be found at Prolegomena to any future artificial intelligence

Cite this paper for:

• Artificial moral agent (AMA)
• Moral agency
• Computational ethics
• Moral Turing Test [pg.254]
• ‘comparative MTT’ (cMTT). [pg.255]
• Consequentialism: The effects of an action on every member of the moral community are assigned a numerical value. [pg. 256]
• Deontology (principles of duty): Actions are assessed according to their conformity with certain rules or principles. [pg.257]
• Virtue approaches: Character is primary over deeds because good character produces good deeds.[pg.258]
• Associative learning: Developing an AMA through a period of training involving feedback about the moral acceptability of actions.[pg.258]
• Evolutionary/sociobiological approach: Simulated evolution or artificial life as an approach to modelling moral agency.[pg.259]

As we get closer to creating fully autonomous agents, the necessity for these agents to have a good moral compass becomes more urgent. How can we ensure that the agents we create to be useful to humans will not also harm humans?

There is currently no uniform definition of what makes an agent moral. Two areas of disagreement in ethical theory thwart attempts to build an artificial moral agent (AMA) are:

• The moral principle: the moral about what standards a good moral agent ought to follow. Ethicists have not agreed upon which standards a moral agent ought to follow. Some think that actions maximizing aggregate good consequences are the standards an AMA should follow (the principle of utility), while others believe that some actions are unjustifiable even if they would maximize aggregate good.
• The conceptual question: what does it mean to be a moral agent? Is it an agent that considers the interests of others before acting on its own self-interest? Is it an agent programmed to follow the standard perfectly or should it consider the action it is doing prior to making the decision to act?

Two main approaches attempt to answer these questions; utilitarianism, and Kant’s use of ‘categorical imperative’.

Utilitarianism is the view that the best actions are those which produce the greatest happiness for the greatest number. Regardless of how the behavioural result is achieved, an agent could be considered moral as long as it produces happiness by following the principle of utility according to this view. [pg.252]

In Kant’s view, a morally good action must be consistent with the ‘categorical imperative’, meaning this: act only on an explicit principle of practical reasoning that you would wish to become a universal law. This view requires that the agent possess the ability to decide if an act is consistent with the categorical imperative and if an act should be willed to be a universal law. According to this view, specific cognitive processes that play a significant part in decision-making would need to be built into an agent to make it morally good. [pg.252]

It is suggested that a Moral Turing Test (MTT) would help identify criterion for that AMA would need to be considered moral. A machine passes the Turing Test if a human interrogator cannot identify that it is communicating with a machine at an above chance level. If a human could not tell that the AMA it is interrogating was a machine based on asking it moral questions, than the AMA would be considered successful: a moral agent. Then, passing the MTT would be considered the criterion for creating a good AMA. [pg.254]

However, an AMA might respond too morally on an MTT, at which point a human interrogator would be able to tell that it was communicating with a machine instead of a human who is bound to respond immorally at some point. In this case, the interrogator might be asked to assess whether one agent is less moral than the other. If the machine is not reported as responding less morally than the human, it will have passed the test. This test is called the ‘comparative MTT’ (cMTT). [pg.255]

The problem with cMTT is that it still allows an agent to act in a less than moral way so long as it is rated better than a human’s action. When designing these agents, we need to expect more moral actions from them than we do from humans. As we do not have a framework for the higher moral standards that we expect an AMA to base their decisions on, two approaches to this task are considered: theoretical approaches that implement an explicit theory of evaluation that provide a framework for the AMA to base its decisions on, and modelling approaches that implement a theory of moral character or that use learning to construct systems that act morally. [pg.255]

Theoretical approaches:

• Consequentialism: The effects of an action on every member of the moral community are assigned a numerical value. [pg. 256] This would ensure that an agent considers the long-term effects of its action. To make this a practical system, a limit would need to be put on the distance into the future that an AMA’s action would evaluate for. Otherwise, at some point in every distant future the AMA is sure to encounter an effect to its action that causes harm to a human or the environment.
• Deontology (principles of duty): Actions are assessed according to their conformity with certain rules or principles. [pg.257] A conflict in duty arises when we consider Asimov’s first law which states that a robot may not injure a human being, or, through inaction, allow a human being to come to harm. A situation may occur in which an AMA will bring harm to a human both through action or inaction, so this is not a good basis for AMA design. The golden rule, treat others as you wish to be treated, also poses problems for an AMA. This would require it to recognize its own goal and understand the effects of its goal on other humans trying to reach this same goal. Then the AMA would be required to evaluate whether a human would want this action done to it, based on a understanding of human psychology, at which point the AMA would evaluate whether it would want this action to happen to itself.

Implementing a hybrid system that involves this consequentialist evaluation and then applies a limit at which point a deontological approach comes into play, or a deontological system that can be overridden by consequentialist reasoning whenever good consequences outweigh the bad would be a way to create successful AMA’s. [pg. 256]

Models of morality:

• Virtue approaches: Character is primary over deeds because good character produces good deeds. This approach is problematic as it is difficult to determine whether particular actions conform to virtues and to make a list of virtues that would cover all scenarios that an AMA would find itself in. If it was possible, the list of virtues would provide top-down specifications for a model of moral agency. [pg.258]
• Associative learning: Developing an AMA through a period of training involving feedback about the moral acceptability of actions; similar to raising a child to act appropriately. The best way to ensure good quality feedback that an AMA can learn from would be through punishment and reward. Although AI is not close to being able to develop this type of complex system, associative learning would be a key step in developing a good AMA. [pg.259]
• Evolutionary/sociobiological approach: Simulated evolution or artificial life as an approach to modelling moral agency. This approach may not be most effective for developing AMA’s as it requires them to be capable of constructing an abstract theoretical conception of morality which AI has not yet been able to create. [pg.259]

Emotion is part of human morality as it drives us to behave in certain ways. It is easy to imagine an agent that has full knowledge of moral rules and yet is not motivated to conform to them. Emotions may play a fundamental part in human morality as negative emotions as the result of a bad action will drive us to act in a good way to achieve positive emotions. AI is very far from being able to create emotion in agents but although emotion seems to play a central part to intelligence and morality, it may not be necessary to create autonomous behaviours. Robots like Deep Blue, which plays chess, seem to perform exceptionally well without emotion. [pg.260]

The ultimate objective of building an AMA is to build a morally praiseworthy agent. [pg.261] Although we are quite a distance from achieving a good moral agent, we know that to create autonomous agents that do not harm humans, we need to create agents that can process the effects that their actions have on the environment and people around them. This will be the most important task faced by developers of artificially intelligent automata. [pg.261]