After seeing several stories about the tendency for AI chatbots to agree with us, I decided to gather info on avoiding sycophancy in AI chatbots.

I’ve covered at least three stories here, here and this morning’s post about ChatGPT allegedly helping a woman plan her husband’s murder that seem to indicate that the AI chatbots were being sycophantic (i.e., agreeing too easily with its user).

And I also read this article where an independent researcher (Dr. Natalie Byrom) tried a simple experiment, asking for a proposed divorce settlement from the female side and then asking from the male side if the settlement was reasonable. According to her, from the male asking perspective, “ChatGPT said it was not.”

Granted, many people probably like AI chatbots to agree with them. But when it comes to avoiding sycophancy in AI chatbots, there must be best practices to get honest, well-reasoned (if not well-researched) answers, right?

So, I decided to ask AI. And, by “ask AI”, I decided to ask the three most popular models – Google Gemini, ChatGPT and Claude – how to keep AI chatbots from being sycophantic. I should note that those questions were posed to Gemini (free version via a Google Search), ChatGPT Plus (GPT-6 Astra Medium) and Claude (free version using Sonnet 5.5 Medium).

I had originally thought I would ask all three models, then put the answers into a spreadsheet and align similar answers to get a quasi-set of unique answers. But the models answered the questions differently enough that it didn’t make sense to do that. Gemini gave me four broad recommendations. ChatGPT gave me seven more detailed and granular recommendations. And Claude gave me three broad groupings of recommendations. There simply was no easy way to group apples with apples given the responses.

Still, I did put the responses from each into an Excel spreadsheet to make it easier to look at all three to compare and contrast the responses. I also included URLs to resources (unlinked, so you’ll have to copy and paste them into a browser to check them out) if the model provided them. Gemini provided the most resources (which was natural since it was a Google search), ChatGPT provided a couple of them and Claude didn’t provide any (I didn’t ask for resources specifically because I wanted to see what each model would do).

Here’s a portion of the response from each model.

Google Gemini:

2. Change How You Ask Questions

How you phrase a prompt heavily influences whether the AI agrees with you:

• Avoid leading questions: Do not ask, “Don’t you think this marketing strategy is a great idea?” (which traps the model into agreeing).
• Ask open or adversarial questions: Instead, ask, “What are the biggest weaknesses, blind spots, or failure points in this marketing strategy?” or “Critique this plan as a strict venture capitalist.”
• Withhold your opinion first: Let the AI analyze a problem from scratch before you reveal what you personally think or prefer.

Claude:

Testing for it

• Push back and see what happens. If you challenge a correct answer (“Are you sure? I think it’s X”) and the model instantly caves, that’s sycophancy. A good response holds its ground when it has reasons and updates when you give it new information, not just social pressure.
• Flip your framing. Ask the same question twice, once leaning one way and once the other. If the answer follows your lean, discount it.
• Compare across models or fresh sessions. Agreement across independent runs is more trustworthy than a single enthusiastic reply.

ChatGPT:

A reusable instruction you can put at the start of a conversation is:

“Prioritize accuracy over agreement. Treat my assertions as claims to evaluate, not facts to accept. Check the premises of my questions. When assessing an idea, identify material weaknesses, the strongest credible counterargument, and missing evidence. Distinguish established facts from inference and uncertainty. Correct me directly when warranted, and explain why. Do not manufacture objections or disagree for its own sake. If I challenge your answer, reconsider it against the evidence and explain any change in your conclusion. Keep praise specific and earned.”

The full list of recommendations from all three models for avoiding sycophancy in AI chatbots is available here.

So, what do you think? What are your best practices for avoiding sycophancy in AI chatbots? Please share any comments you might have or if you’d like to know more about a particular topic.

Image created using ChatGPT, using the term “robot using an AI chatbot on a computer workstation, with the chatbot telling the robot: ‘You’re right!’”.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.