Patrick Butlin explores the idea of whether these systems could possess the capacity to “act for reasons”, a concept traditionally associated with conscious and goal-directed agents. Drawing upon philosophical literature and specifically from the work of Hanna Pickard (2015) and Helen Steward (2012), the author outlines two criteria needed to be met for something to be considered an agent: the entity in question must have goals and it must also interact with its environment to pursue these goals. The author asserts that both model-free and model-based RL systems meet these criteria and can thus be considered as having minimal agency.
Building upon the foundation of minimal agency, the author makes a compelling argument for RL systems acting for reasons. Their argument hinges on the philosophical work of Jennifer Hornsby (2004) and Nora Mantel (2018), where the former associates acting for reasons with general-purpose abilities, and the latter distinguishes between three competencies involved in action for reasons: epistemic, volitional, and executive sub-competences. The author posits that model-based RL systems, with their capacity to model the transition function, meet these criteria as they learn and store information about their environment that influences their future actions, forming a sort of ‘descriptive representation’.
In contrast to Mantel, the author suggests that the distinction between volitional and executive sub-competences and the emphasis on motivation might not be necessary to the account. While Mantel uses motivation interchangeably with desire and intention, the author posits that this distinction might be more relevant to human agency and less so for artificial RL systems. The author also refutes the notion that the lack of desires or volitions disqualifies artificial RL systems from acting for reasons. They conclude that while model-based RL systems may lack desires, their interaction with their environment to achieve set goals provides sufficient grounds to attribute minimal agency to them and thus the capacity to act for reasons.
The article adds significantly to the discourse on machine agency, challenging conventional philosophical norms that tie agency and the capacity to act for reasons to consciousness or biological entities. It raises compelling points about how RL systems, through their goal-directed behavior and interaction with the environment, exhibit traits of minimal agency. This exploration places the discussion of machine agency within broader philosophical themes such as the nature of consciousness, the demarcation of human and non-human agency, and the implications of attributing agency to artificial systems.
Future research could focus on extending the arguments in this article, exploring the implications of attributing even more sophisticated forms of agency to artificial RL systems. One direction could be to look at whether these systems, as they continue to develop, could eventually meet even stricter criteria for agency that go beyond minimal agency. Another avenue would be to study the ethical and societal implications of recognizing artificial RL systems as agents. Would it, for instance, be meaningful or necessary to establish an ethical framework for interacting with these systems? Additionally, research could examine how these concepts might evolve in tandem with the continued development of artificial RL systems and other forms of artificial intelligence.
Abstract
There is an apparent connection between reinforcement learning and agency. Artificial entities controlled by reinforcement learning algorithms are standardly referred to as agents, and the mainstream view in the psychology and neuroscience of agency is that humans and other animals are reinforcement learners. This article examines this connection, focusing on artificial reinforcement learning systems and assuming that there are various forms of agency. Artificial reinforcement learning systems satisfy plausible conditions for minimal agency, and those which use models of the environment to perform forward search are capable of a form of agency which may reasonably be called action for reasons.
Reinforcement learning and artificial agency