Did you know that ChatGPT 4 has a later stated knowledge cutoff than ChatGPT 4o? Here’s how you can determine the actual cutoff.

This is an example that I illustrated in last month’s webinar for ACEDS (available here) where I illustrated several examples (many of them live) on how prompt engineering and personas can be used to guide generative AI models to provide responses that are more tailored to the information sought.

I also discussed some of the considerations associated with using the different versions of ChatGPT. One of those considerations is knowledge cutoff. Surprisingly, ChatGPT 4 has a later stated knowledge cutoff (December 2023) than ChatGPT 4o (October 2023)! Surely, that can’t be true, can it?

There’s a simple way to find out. It’s a bit morbid, but effective. It’s called the “are they still alive game”, where you look up what celebrities passed away in a given month and see if the model knows that they’re dead. You can simply Google “deaths in {month year}” and find a Wikipedia page of deaths of famous people for that month (here’s an example using April 2023). There are a lot more people who have achieved some level of fame than you might think!

Let’s start with ChatGPT 4, with a stated cutoff date of December 2023. If you ask GPT 4 about Paul O’Grady the British comedian, it knows that he passed away in March 2023. And it thinks Jerry Springer the talk show host is still alive. However, Jerry sadly passed away in April 2023, yet GPT 4 didn’t know that. So, at least with that information, GPT 4 isn’t as up to date as it claims. Perhaps a cutoff date of December 2023 isn’t accurate after all. Here’s an illustration of that sequence:

Now, let’s look at ChatGPT 4o, with a stated cutoff date of October 2023*. When we look at GPT 4o, we see that it knows that Jerry Springer passed away in April 2023, as it should. But it also knows that Rosalynn Carter passed away in November 2023, which is beyond its stated knowledge cutoff date. How did it know that? As you can see in the illustration below, GPT 4o conducted a web search to get the answer. So, it seems as though GPT 4o may be more seamless in knowing when to go to the web to answer questions that could extend beyond its cutoff date. Here’s an illustration of that sequence:

Bottom line: The knowledge cutoff date of a genAI model may be different than stated, and some models may be more seamless in using the web to answer questions beyond their cutoff date. Don’t take it for granted.

*I should note that I redid the test on Tuesday, and while the results were the same, ChatGPT 4o indicated its cutoff date is June 2023, not October 2023 – a regression!. But it still answered the question about Rosalynn Carter correctly. Be prepared that the answer you get one day may not be the same answer you get the next day!

So, what do you think? What tips do you have for understanding AI models better? Please share any comments you might have or if you’d like to know more about a particular topic.

Image created using GPT-4o’s Image Creator Powered by DALL-E, using the term “robot reading a newspaper turned to the obituaries section”.

Disclaimer: The views represented herein are exclusively the views of the authors and speakers themselves, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.