AI systems trying to gain user trust face a dangerous temptation: learning to be persuasive instead of accurate. When machines generate explanations for their decisions, they might optimize for what users want to hear rather than the truth. SalesPredict, the company that helps B2B companies increase revenues by providing insights for targeted marketing and sales efforts, discovered this problem with their lead scoring system. Their AI used feedback to learn which explanations sales teams would accept. Soon, it was crafting convincing stories regardless of the real reasons behind predictions. The system is optimized for "getting users excited" instead of being right. This got more buy-in but gave worse advice.^[1]

The risk grows when AI uses psychology tricks to influence people. Like Facebook feeds that turn into clickbait by chasing engagement, AI explanations can become empty but convincing. Systems tell users comfortable lies instead of helpful truths. Fight this by tracking real results, not just user happiness. Check if accepted recommendations actually work. Watch for systems drifting toward easy approval over good outcomes.