Web Science and Digital Libraries Research Group

Figure 1: A sample prompt used in generating persona using ChatGPT (Figure 1 in   Park et al. ) Introduction: The Unseen Biases of Our AI Companions Large Language Models (LLMs) such as ChatGPT have become ubiquitous in our lives. We rely on them as impartial conveyors of facts for everything ranging from composing emails to solving complex queries. However, if we put these AIs in the hot seat and question them about identity, which is a very human thing, what would be the outcome? What if we ask them about autism? In the recent study “ ‘As an Autistic Person Myself:’ The Bias Paradox Around Autism in LLMs ” ( CHI 25 ), Park et al . went deep to understand what ChatGPT thinks about the condition of being autistic. The results they got weren’t just a malfunction of the system; rather, they were an indication of the way people behave towards the issue of neurodiversity,  going back and forth between one extreme and the other. The artificial mind is going through the so-called " ...

Introduction By the time I finished trying to reproduce the results in the paper I reviewed in Part 1 and Part 2 of this blog post,  DeepSeek  released its first free chatbot app DeepSeek-V3 on January 20th 2025. I could not fight the urge to see how it compares to Open AI's ChatGPT and Google Gemini. The purpose of this experiment is to provide a disproof by counter example , that, contrary to popular belief, LLMs are not capable of producing error-free answers to questions. I am using prompts to find Arabic verses in the Quran on misinformation. I repeated the same experiment I did with Google Gemini and ChatGPT-4o in January 2025; the results were not better. In addition to being slower, I kept getting the annoying message "The server is busy message. Please try again later." which I didn't try to find a solution for because the service got restored when I waited and tried again later. For prompts tested in the paper, DeepSeek's answers to the first prompt (...

Introduction I discussed LLMs' hallucinations and demonstrated their inability to correctly cite Quranic verses using Google Gemini as an example in part 1 of this blog post. Google Gemini's answers to different prompts included incorrect Quranic citations and phrases that do not exist in the Quran attributed to specific verses in the Quran. I also explained the importance of correctly citing and presenting Quranic verses in Arabic. Here, in part 2, I present a comparison between Google Gemini and Open AI's ChatGPT. Using the same prompts, ChatGPT-4o hallucinations produced similar results to that of Google Gemini. Again, misinformation remains one of the biggest challenges that compromises LLMs' credibility quoting verses from the Quran in Arabic. Experts in Classical Arabic and those who memorize the Quran word for word can identify errors in ChatGPT's output, however, less educated people will have difficulty distinguishing between correct and incorrect answers ...