On the ethics of emerging technologies, Marcello Ienca critically examines the role of digital technologies, particularly artificial intelligence, in facilitating manipulation. This research involves a comprehensive analysis of the nature of manipulation, its manifestation in the digital realm, impacts on human agency, and the ethical ramifications thereof. The findings illuminate the nuanced interplay between technology, manipulation, and ethics, situating the discussion about technology within the broader philosophical discourse.
Ienca distinguishes between concepts of persuasion and manipulation, underscoring the role of rational defenses in bypassing effective manipulation. Furthermore, they unpack how artificial intelligence and other digital technologies contribute to manipulation, with a detailed exploration of tactics such as personalization, emotional appeal, social influence, repetition, trustworthiness, user awareness, and time constraints. Finally, they propose a set of mitigation strategies, including regulatory, technical, and ethical approaches, that aim to protect users from manipulation.
The Nature of Manipulation
Within the discourse on digital ethics, the issue of manipulation has garnered notable attention. Ienca begins with an account of manipulation, revealing its layered complexity. They distinguish manipulation from persuasion, contending that while both aim to alter behavior or attitudes, manipulation uniquely bypasses the rational defenses of the subject. They posit that manipulation’s unethical nature emerges from this bypassing, as it subverts the individual’s autonomy. While persuasion is predicated on providing reasons, manipulation strategically leverages non-rational influence to shape behavior or attitudes. The author, thus, highlights the ethical chasm between these two forms of influence.
Building on this, the author contends that manipulation becomes especially potent in digital environments, given the technological means at disposal. Digital technologies, such as AI, facilitate an unprecedented capacity to bypass rational defenses by harnessing a broad repertoire of tactics, including personalized messaging, emotional appeal, and repetition. These tactics, which exploit the cognitive vulnerabilities of individuals, are coupled with the broad reach and immediate feedback afforded by digital platforms, magnifying the scope and impact of manipulation. As such, Ienca’s research contributes to a deeper understanding of the nature of digital manipulation and its divergence from the concept of persuasion.
Digital Technologies and the Unraveling of Manipulation
Ienca critically engages with the symbiotic relationship between digital technologies and manipulation. They elucidate that contemporary platforms, such as social media and search engines, employ personalized algorithms to curate user experiences. While such personalization is often marketed as enhancing user satisfaction, the author contends it serves as a conduit for manipulation. These algorithms invisibly mould user preferences and beliefs, thereby posing a potent threat to personal autonomy. The authors extend this analysis to AI technologies as well. A key dimension of their argument is the delineation of “black-box” AI systems, which make decisions inexplicably, leaving users susceptible to undisclosed manipulative tactics. The inability to scrutinize the processes underpinning these decisions amplifies their potential to manipulate users. The author’s analysis thus illuminates the subversive role digital technologies play in exacerbating the risk of manipulation, informing a nuanced understanding of the ethical complexities inherent to digital environments.
Ienca posits that such manipulation essentially thrives on two key elements – informational asymmetry and cognitive bias exploitation. Informational asymmetry is established when the algorithms controlling digital environments wield extensive knowledge about the user, engendering a power imbalance. This understanding is used to shape user experience subtly, enhancing the susceptibility to manipulation. The exploitation of cognitive biases further solidifies this manipulation by capitalizing on inherent human tendencies, thus subtly directing user choices. An example provided is the use of default settings, which exploit the status quo bias and contribute to passive consent, a potent form of manipulation. The author’s exploration of these elements illustrates the insidious mechanisms by which digital manipulation functions, enriching our understanding of the dynamics at play within digital landscapes.
Mitigation Strategies for Digital Manipulation and the Broader Philosophical Discourse
Ienca proposes a multi-pronged strategy to curb the pervasiveness of digital manipulation, relying significantly on user education and digital literacy, contending that informed users can better identify and resist manipulation attempts. Transparency, particularly around the use of algorithms and data processing practices, is also stressed, facilitating users’ understanding of their data’s utilization. From a regulatory standpoint, the authors discuss the role of governing bodies in enforcing laws that protect user privacy and promote transparency and accountability. The EU AI Act (2021) is highlighted as a significant stride in this direction. The authors also advocate for ethical design, suggesting that prioritizing user cognitive liberty, privacy, transparency, and control in digital technology can reduce manipulation potential. They also highlight the potential of policy proposals aimed at enshrining a neuroright to cognitive liberty and mental integrity. In their collective approach, Ienca and Vayena synthesize technical, regulatory, and ethical strategies, underscoring the necessity of cooperation among multiple stakeholders to cultivate a safer digital environment.
This study on digital manipulation connects to a broader philosophical discourse surrounding the ethics of technology and information dissemination, particularly in the age of proliferating artificial intelligence. It is situated at the intersection of moral philosophy, moral psychology, and the philosophy of technology, inquiring into the agency and autonomy of users within digital spaces and the ethical responsibility of technology designers. The discussion on ‘neurorights’ brings to the fore the philosophical debate on personal freedom and cognitive liberty, reinforcing the question of how these rights ought to be defined and protected in a digitized world. The author’s consideration of manipulation, not as an anomaly, but as an inherent characteristic of pre-designed digital environments challenges traditional understanding of free will and consent in these spaces. This work contributes to the broader discourse on the power dynamics between technology users and creators, a topic of increasing relevance as AI and digital technologies become ubiquitous.
Abstract
The increasing diffusion of novel digital and online sociotechnical systems for arational behavioral influence based on Artificial Intelligence (AI), such as social media, microtargeting advertising, and personalized search algorithms, has brought about new ways of engaging with users, collecting their data and potentially influencing their behavior. However, these technologies and techniques have also raised concerns about the potential for manipulation, as they offer unprecedented capabilities for targeting and influencing individuals on a large scale and in a more subtle, automated and pervasive manner than ever before. This paper, provides a narrative review of the existing literature on manipulation, with a particular focus on the role of AI and associated digital technologies. Furthermore, it outlines an account of manipulation based of four key requirements: intentionality, asymmetry of outcome, non-transparency and violation of autonomy. I argue that while manipulation is not a new phenomenon, the pervasiveness, automaticity, and opacity of certain digital technologies may raise a new type of manipulation, called “digital manipulation”. I call “digital manipulation” any influence exerted through the use of digital technology that is intentionally designed to bypass reason and to produce an asymmetry of outcome between the data processor (or a third party that benefits thereof) and the data subject. Drawing on insights from psychology, sociology, and computer science, I identify key factors that can make manipulation more or less effective, and highlight the potential risks and benefits of these technologies for individuals and society. I conclude that manipulation through AI and associated digital technologies is not qualitatively different from manipulation through human–human interaction in the physical world. However, some functional characteristics make it potentially more likely of evading the subject’s cognitive defenses. This could increase the probability and severity of manipulation. Furthermore, it could violate some fundamental principles of freedom or entitlement related to a person’s brain and mind domain, hence called neurorights. To this end, an account of digital manipulation as a violation of the neuroright to cognitive liberty is presented.
On Artificial Intelligence and Manipulation