A leading AI philosopher has issued a sharp warning against one of the most common assumptions in artificial intelligence ethics, that the goal of developers should be to make users trust AI systems. In a new paper, Dr. Giacomo Zanotti of Politecnico di Milano argues that this approach risks creating false confidence in machines, eroding human oversight, and undermining the very principles that define responsible AI.
Published in AI & Society and titled “AI Systems Should Be Trustworthy, Not Trusted,” the study dismantles the growing obsession with “trust in AI” and redirects attention to what Zanotti calls the only legitimate ethical goal: making AI systems trustworthy. His analysis draws on philosophical theory, empirical research, and regulatory frameworks such as the EU’s Ethics Guidelines for Trustworthy AI and the AI Act, proposing that the future of ethical AI depends on fostering continuous vigilance rather than uncritical trust.
The misplaced focus on “Trust in AI”
The first argument challenges the foundation of today’s AI governance discourse, the belief that human trust should be cultivated as a design and policy objective. The author makes a clear distinction between trust, which is a psychological and emotional response, and trustworthiness, which is an objective property of a system.
Trust, he explains, is not a reliable indicator of safety or ethics because it is shaped by emotions, familiarity, and appearance rather than evidence. People routinely trust systems that are flawed or untested simply because they look competent or behave in human-like ways. Conversely, they may distrust systems that are genuinely reliable if they lack intuitive or emotional cues.
This cognitive mismatch, the author argues, exposes a central problem in AI ethics: the assumption that “trust” can be engineered as a design feature. He points to the widespread use of design strategies that encourage users to bond with machines, friendly chat interfaces, empathetic voice tones, or reassuring visual cues. These techniques increase comfort and engagement but do not guarantee that the system itself is safe, unbiased, or transparent.
The focus on trust is therefore not only misguided but potentially dangerous. By centering human psychology rather than objective integrity, developers risk confusing perception with performance. The ethical responsibility, Zanotti insists, lies not in persuading users to feel secure but in ensuring that systems behave reliably under scrutiny.
This distinction aligns with the European Union’s legal framework for AI, which prioritizes accountability, transparency, and robustness as measurable components of trustworthiness. In contrast, trust as an emotion cannot be standardized or verified, making it an unstable foundation for governance.
The ethical risks of designing for trust
The study also focuses on how designers attempt to make AI systems appear “trustworthy” through anthropomorphism, the practice of giving machines human-like traits, emotions, or voices. While this technique may foster user comfort, Zanotti shows that it also manipulates social instincts, blurring ethical boundaries.
Voice assistants, social robots, and chatbots are now engineered to mimic empathy and warmth. Research shows that users respond to these cues as they would to human interactions, developing a sense of rapport or emotional attachment. However, this response arises from psychological reflexes, not rational evaluation. A digital assistant that says “I’m sorry” after an error may appear responsible, but such gestures are performative, not ethical.
Zanotti argues that anthropomorphic design creates an illusion of moral agency. By making systems seem caring or self-aware, developers encourage users to suspend critical judgment. This artificial emotional connection can lead to overreliance on AI and diminish awareness of its limitations.
More critically, this design philosophy conflicts with the spirit of trustworthy AI, as defined by the EU’s 2019 ethical guidelines and reaffirmed by the 2024 AI Act. These frameworks explicitly prohibit manipulative behavior, including deceptive representations of machines as human. Designing for emotional appeal, even indirectly, risks turning AI systems into tools of persuasion rather than instruments of accountability.
The ethical issue extends beyond user manipulation. When trust is treated as a product, companies may prioritize perception management over technical rigor. Systems become optimized for likability, not reliability. Zanotti warns that this approach undermines long-term public confidence and shifts focus from governance to branding.
Instead of making users trust AI, he calls for developers to make systems that can be trusted upon verification, systems whose architecture, documentation, and decision logic withstand scrutiny by experts, regulators, and users alike. Trust should follow evidence, not interface design.
The paradox of trust and oversight
The author’s most provocative argument focuses on a philosophical paradox: trust and monitoring cannot coexist. When people trust someone, they stop checking. Translated to AI, this means that encouraging trust naturally discourages oversight, precisely the opposite of what responsible AI demands.
The paper draws from philosophical theories of trust, from Annette Baier’s notion of trust as a “fragile plant” that withers under suspicion to Nguyen’s framing of trust as an “unquestioning attitude.” In both views, trust reduces vigilance. This becomes problematic in the context of AI systems that require constant supervision.
Modern AI, particularly those based on deep learning and adaptive algorithms, operate in uncertain and dynamic environments. Their outputs evolve with data, meaning that their behavior cannot be predicted or validated once and for all. The EU AI Act acknowledges this by mandating post-market monitoring, a continuous evaluation of system performance, bias, and safety after deployment.
The study argues that if users or operators begin to trust these systems, they may gradually reduce monitoring. The more the system is trusted, the less it is watched. In safety-critical sectors such as healthcare, aviation, or autonomous transport, this complacency can lead to catastrophic failures.
This insight exposes the contradiction at the heart of the “trustworthy AI” movement: encouraging trust in systems that must, by design, remain under constant human oversight. True ethical AI requires critical vigilance, not emotional reassurance.
Zanotti illustrates this risk with real-world parallels where automation bias, the tendency to over-rely on machine outputs, has led to errors. In medical diagnostics, for example, doctors who defer excessively to algorithmic predictions may overlook contextual cues or anomalies. Similar patterns appear in automated credit scoring, predictive policing, and navigation systems.
The solution, the author proposes, is a cultural shift from trust to accountable dependence. Humans will inevitably rely on AI, but reliance must remain conscious, monitored, and reversible. This reframing aligns with the core principles of trustworthy AI, fairness, transparency, and human oversight, while rejecting the psychological trap of unearned confidence.
A framework for ethical AI development
The study does not dismiss the concept of “trustworthy AI” altogether; rather, it restores its meaning by grounding it in objective qualities rather than emotional states. A trustworthy AI system, according to Zanotti, must meet verifiable standards:
- Reliability and robustness: The system performs consistently across environments and data variations.
- Transparency: Its decision-making process and limitations are clear to users and regulators.
- Ethical alignment: It respects human rights, dignity, and autonomy in design and function.
- Accountability: It includes mechanisms for traceability, auditability, and error correction.
- Sustainability: It supports societal and environmental well-being.
The author rejects the claim that only humans can be trustworthy. While AI lacks moral agency, systems can embody trustworthiness through ethical design, governance, and regulation. This approach aligns with international standards such as ISO/IEC 24028:2020 and ISO/IEC 42001:2024, which define metrics for AI robustness and accountability.
For policymakers, this means shifting evaluation criteria from how much users trust AI to how well AI systems earn trust through evidence. For designers, it means prioritizing function and transparency over aesthetics and emotion. For society, it means cultivating a culture of informed skepticism, a public capable of engaging critically with the systems it uses.
The framework extends to developers and ethicists alike: avoid creating AI that persuades; build AI that withstands scrutiny. Ethical technology should neither demand trust nor exploit it but continually prove its integrity through openness and accountability.
