AI Chatbots Rely on Sources With Clear Biases

AI chatbots like ChatGPT and Grok can be a big help in writing essays, conducting research, and exploring complex issues. But these tools bring risks, especially when they filter facts through a political lens. And the Trump administration is now stepping into the debate. “We believe AI systems should operate free of ideological bias and avoid pushing socially engineered agendas,” said David Sacks, the administration’s AI and crypto czar, in a statement today. “We’ve introduced several proposals to ensure AI stays truth-seeking and trustworthy.”

Over the weekend, I saw this bias unfold in real time.

The Grok chatbots turn gun-control advocate

On Friday, a user on Elon Musk’s platform X asked Grok whether more guns make Americans safer. Grok responded flatly: “No, evidence shows more guns correlate with higher firearm homicides and violent crime rates.” The chatbot dismissed self-defense and deterrence, referring to my research –specifically my “more guns, less crime” theory – as something cited by “right-wing advocates.” Grok supported its claims by referencing Scientific American magazine and a RAND Corporation review, saying these sources show guns don’t reduce crime and instead increase violence.

Those answers are misleading and wrong.

The Scientific American article had extensive biases. Grok ignored my published rebuttal in Scientific American. In it, I noted that over two-thirds of peer-reviewed studies show that concealed carry laws do reduce crime. Melinda Wenner Moyer, a journalist affiliated with Michael Bloomberg’s The Trace, a well-known gun control advocacy outlet, wrote the article. I had provided Moyer with those studies while she prepared her piece, but she ignored them. She failed to acknowledge any of my post-1998 work and misrepresented the findings of the National Research Council’s major report on the topic.

Moving the goalposts

Grok gave tremendous weight to RAND’s literature survey, claiming that RAND had surveyed 100+ studies. Eventually, Grok conceded that the number of papers studying right-to-carry laws was actually 25, showing a range of mixed results. I pointed out that the California-based think tank was highly selective in the sources it included, ignoring dozens more papers showing that these laws lowered violent crime rates and surveys of academics who have published peer-reviewed empirical research.

Even then, Grok largely ignored my responses and focused on two papers claiming that right-to-carry laws increased violent crime. The first failed to control for any variables – such as changes in policing, poverty, or economic conditions – that affect crime trends after adopting right-to-carry laws. When I pointed that out, Grok mentioned another study that demonstrated a statistical technique that could account for such factors, but that study didn’t look at right-to-carry laws. Only after a prolonged exchange did Grok acknowledge the error.

Comparing ripe apples to non-ripe apples

The second paper Grok emphasized made a subtler mistake: It compared states that had recently adopted right-to-carry laws to states that had adopted such laws years earlier. The early adopters made it easier to obtain permits and saw much larger increases in concealed handgun permits during the period studied. Comparing later adopters – who saw smaller increases – to these early states skewed the results. If crime didn’t fall as much in the newer states, the flawed analysis made it look as if crime had risen. Again, only after I cited my own peer-reviewed studies from 2022 and 2024 did Grok acknowledge the issue.

When Grok argued that more guns lead to more firearm homicides, I asked it to name any country that banned all guns or handguns and saw homicide rates fall – or even stay the same.

Three countries where gun control did not work as advertised

Grok cited Australia, Great Britain, and Brazil, but none of those examples are accurate.

Australia never banned all guns or handguns. Firearm homicides had already been falling for 15 years before the 1997 buyback and fell more slowly afterward. Meanwhile, gun ownership actually increased and by 2010 had surpassed pre-buyback levels.

In Britain, handgun bans enacted in 1997 preceded a 50% surge in homicide rates over the next seven years. The rates didn’t decline until the government boosted the police force by 14% over two years. Even then, homicide rates took 14 years to return to pre-ban levels.

Brazil didn’t ban all guns or handguns either. While its 2003 gun control law included a boost in law enforcement resources, murder rates remained largely unchanged. Only after President Jair Bolsonaro took office in January 2019 – liberalizing gun ownership and increasing legal gun ownership by 650% – did Brazil’s homicide rate drop by more than 30%.

Only after I laid out these facts did Grok concede, calling them “fair points” and then echoing the very arguments I had just made.

Chatbots tilt left and rely on biased sources

My experience with Grok is not unique. To study the chatbots’ political biases, the Crime Prevention Research Center, which I head, asked various AI programs questions on crime and gun control last year in March and again in August and ranked the answers on how progressive or conservative their responses were. The chatbots tilted to the left, claiming that things like higher arrest and conviction rates don’t deter crime and clearly supporting more gun control laws.

AI chatbots speak with certainty but often rely on sources with clear biases. They cite selective evidence, misrepresent or don’t understand complex findings, and ignore reputable research that challenges a politically convenient narrative. AI chatbots also hallucinate, meaning they sometimes completely make up facts.

Students, journalists, and everyday citizens increasingly rely on these tools. If they accept chatbot responses at face value, they risk walking away with a fundamentally distorted view of issues like gun policy.

This article was originally published by RealClearPolitics and made available via RealClearWire.