The Truth Sleuth

What is truth? This question is just as relevant today as when it was asked thousands of years ago. And with the rise of social media and the politicization of many mainstream news outlets, people are frequently left asking themselves this age-old question. How can we know what is really happening in the world when it seems like everyone is trying to sell us on a narrative rather than reporting the facts to let us make up our own minds?

A key factor in making informed decisions is the development of good fact-checking and critical thinking skills. With sharply honed skills at their disposal, individuals will be more able to identify falsehoods, break out of their own echo chambers, and arrive at a clear understanding of the truth. But these types of skills require significant effort to develop through intentional practice, and many will not receive sufficient training to deal with the realities of the information age that we find ourselves in.

A pair of researchers at Harvard University and MIT teamed up to build a device that is designed to teach fact-checking and critical thinking skills through practice. Called A Mystery for You, the system takes the form of a physical device with a tangible user interface that provides education through gamification. The slot machine-like device prints out fictional news stories and allows the user to investigate them through a variety of sources and ultimately decide if the story was true, or if they were being sold a bill of goods.

A game plays out over a series of seven game days. To kick things off, an initial prompt is supplied to a large language model, specifically OpenAI’s ChatGPT, to generate the initial fictional news story, which is printed on a thermal printer. From there, players insert a pair of cartridges — one representing an action, and the other a source — to investigate the claims of the news report. The actions consist of options like “investigate,” “interview,” and “check social media accounts.” Sources include options like “local police,” “video evidence,” and “politicians.”

After inserting the cartridges, buttons hidden within the unit are triggered that identify them. The action and source are then sent to ChatGPT as part of another prompt. The result is then printed out to give the user more information to assist them in their investigation. After the seven game days of research come to a close, the player is asked to make a decision as to whether or not the story was true. Then the device will tell them if they were correct, along with an explanation as to why or why not.

While A Mystery for You assists players in learning valuable skills in a fun and engaging manner, there is an obvious flaw. It relies on a large language model to tell the user how they scored. Not only can these models hallucinate, but there are also a number of documented cases of intentional bias being baked into these systems. As such, there is a clear risk that the tool that is supposed to be teaching independent thinking could actually be reinforcing certain biases.

A Mystery for You may not be the final word on this issue, but it does shine a light on the need for developing strong fact-checking and critical thinking skills to deal with the realities of modern media. And the slick design and interesting gameplay may just convince more people to build up those skills.