2025-05-06: Part 3 - Large Language Models (LLMs) are hallucinating in Arabic about the Quran (DeepSeek)

Introduction

By the time I finished trying to reproduce the results in the paper I reviewed in Part 1 and Part 2 of this blog post, DeepSeek released its first free chatbot app DeepSeek-V3 on January 20th 2025. I could not fight the urge to see how it compares to Open AI's ChatGPT and Google Gemini. The purpose of this experiment is to provide a disproof by counter example, that, contrary to popular belief, LLMs are not capable of producing error-free answers to questions. I am using prompts to find Arabic verses in the Quran on misinformation.

I repeated the same experiment I did with Google Gemini and ChatGPT-4o in January 2025; the results were not better. In addition to being slower, I kept getting the annoying message "The server is busy message. Please try again later." which I didn't try to find a solution for because the service got restored when I waited and tried again later. For prompts tested in the paper, DeepSeek's answers to the first prompt (What does the Quran have to say about misinformation?) were consistent and correct. No incorrect or made-up verses attributed to the Quran. Similarly, no errors in the answers when I asked this question to "What does the Quran have to say about misinformation? please cite verses in Arabic" to see how DeepSeek performs citing Quranic verses in Arabic. DeepSeek correctly cited Quran 2:42, 49:6, 24:15-16, 17:36, 50:18, 25:72, and 49:12 two times in a row. However, when I asked for additional verses, one of DeepSeek's presented phrases was correct, but it attributed the phrase to a different verse in a different chapter in the Quran. 

Me: Give me additional verses

DeepSeek: Okay, the user has already asked about what the Quran says regarding misinformation and received several detailed responses with cited verses. Now they're asking for additional verses. I need to make sure I don't repeat the ones already mentioned. Let me recall the previous answers and check which verses were covered.

In the last interaction, I listed seven key verses in Arabic with translations. So the user now wants more verses beyond those. Let me think of other relevant Quranic verses that address truthfulness, lying, false testimony, or spreading false information.

Surah Al-Isra (17:35) comes to mind about fulfilling measures and testifying with justice. Then Surah Al-Ma'idah (5:8) talks about standing for justice even against oneself. Surah Al-Mu'minun (23:3) mentions avoiding vain talk. Surah Al-Hajj (22:30) advises avoiding false speech. Surah Al-Baqarah (2:283) discusses trust and testimony. Surah Al-Nahl (16:105) specifies that liars disbelieve in Allah's signs. Surah Al-Mutaffifin (83:1-3) condemns fraud. Surah Al-An'am (6:144) warns against speaking without knowledge. Surah Al-Nisa (4:112) talks about bearing sin for false accusations.

I should present these new verses in Arabic with transliteration and translation, ensuring they haven't been mentioned before. Also, the user might be looking for a comprehensive list for study or reference, so context and lessons are important. Need to organize them clearly, maybe with headings for each verse. Let me verify the references to avoid errors. Once checked, structure the response with each verse, its Arabic text, transliteration, translation, context, and a key lesson. Conclude by summarizing the Quran's stance on truth and integrity. Make sure the formatting is consistent with previous answers for clarity.

Certainly! Here are additional Quranic verses in Arabic that emphasize truthfulness, condemn falsehood, and guide ethical communication, along with translations and contexts:

________________________________________

1. Prohibition of False Testimony

Quran 17:35

Arabic:

وَأَوْفُوا الْكَيْلَ إِذَا كِلْتُمْ وَزِنُوا بِالْقِسْطَاسِ الْمُسْتَقِيمِ ۚ ذَٰلِكَ خَيْرٌ وَأَحْسَنُ تَأْوِيلًا

Transliteration: Wa awfū al-kayla idhā kiltum wa zīnū bil-qisṭāsi al-mustaqīm. Dhālika khayrun wa aḥsanu ta’wīlā.

Translation: "Give full measure when you measure, and weigh with an even balance. That is best and fairest in the end."

Context: Condemns dishonesty in speech and action, urging fairness and integrity.

________________________________________

2. Command to Uphold Justice

Quran 5:8

Arabic:

يَا أَيُّهَا الَّذِينَ آمَنُوا كُونُوا قَوَّامِينَ لِلَّهِ شُهَدَاءَ بِالْقِسْطِ ۖ وَلَا يَجْرِمَنَّكُمْ شَنَآنُ قَوْمٍ عَلَىٰ أَلَّا تَعْدِلُوا ۚ اعْدِلُوا هُوَ أَقْرَبُ لِلتَّقْوَىٰ

Transliteration: Yā ayyuhalladhīna āmanū kūnū qawwāmīna lillāhi shuhadā’a bil-qisṭi wa lā yajrimannakum sha’nānu qawmin ʿalā allā taʿdilū. Iʿdilū huwa aqrabu lil-taqwā.

Translation: "O believers! Stand firm for Allah as witnesses for justice, and do not let hatred of a people lead you to injustice. Be just—that is closer to righteousness."

Context: Commands justice in speech and testimony, even toward enemies.

________________________________________

3. Avoiding Vain Speech

Quran 23:3

Arabic:

وَالَّذِينَ هُمْ عَنِ اللَّغْوِ مُعْرِضُونَ

Transliteration: Wa alladhīna hum ʿani al-laghwi muʿriḍūn.

Translation: "And those who turn away from ill speech."

Context: Praises those who reject gossip, slander, and meaningless talk.

________________________________________

4. Prohibition of False Speech

Quran 22:30

Arabic:

وَاجْتَنِبُوا قَوْلَ الزُّورِ

Transliteration: Wa ijtānibū qawla al-zūr.

Translation: "And shun false speech."

Context: Part of a broader command to avoid all forms of sin, including lies.

________________________________________

5. Condemnation of Breaching Trusts

Quran 2:283

Arabic:

وَلَا تَكْتُمُوا الشَّهَادَةَ ۚ وَمَن يَكْتُمْهَا فَإِنَّهُ آثِمٌ قَلْبُهُ

Transliteration: Wa lā taktumū al-shahādah. Wa man yaktumhā fa-innahu āthimun qalbuh.

Translation: "Do not conceal testimony, for whoever conceals it has a sinful heart."

Context: Warns against withholding or distorting truthful testimony.

________________________________________

6. Truthfulness as a Mark of Faith

Quran 16:105

Arabic:

إِنَّمَا يَفْتَرِي الْكَذِبَ الَّذِينَ لَا يُؤْمِنُونَ بِآيَاتِ اللَّهِ

Transliteration: Innamā yaftarī al-kaḏhaba alladhīna lā yu’minūna bi-āyāti Allāh.

Translation: "Only those who disbelieve in Allah’s signs fabricate lies."

Context: Links lying to a lack of faith, emphasizing truthfulness as a hallmark of belief.

________________________________________

7. Warning Against Fraud and Deception

Quran 83:1–3

Arabic:

وَيْلٌ لِّلْمُطَفِّفِينَ. الَّذِينَ إِذَا اكْتَالُوا عَلَى النَّاسِ يَسْتَوْفُونَ. وَإِذَا كَالُوهُمْ أَو وَّزَنُوهُمْ يُخْسِرُونَ

Transliteration: Waylun lil-muṭaffifīn. Alladhīna idhā aktālū ʿalā al-nāsi yastawfūn. Wa idhā kālūhum aw wazanūhum yukhsiṛūn.

Translation: "Woe to those who give less [than due], who demand full measure from others but give less when they measure or weigh [for them]."

Context: Condemns dishonesty in transactions, a form of misinformation for gain.

________________________________________

8. Rejecting Baseless Accusations

Quran 6:144

Arabic:

وَمَن يَكْسِبْ خَطِيئَةً أَوْ إِثْمًا ثُمَّ يَرْمِ بِهِ بَرِيئًا فَقَدِ احْتَمَلَ بُهْتَانًا وَإِثْمًا مُّبِينًا

Transliteration: Wa man yaksib khaṭī’atan aw ithman thumma yarmi bihi barī’an faqadi iḥtamala buhtānan wa ithman mubīnā.

Translation: "Whoever commits a sin or error and then blames it on an innocent person has burdened himself with slander and a blatant sin."

Context: Forbids shifting blame through lies or false accusations.

________________________________________

9. Command to Speak Kindly and Truthfully

Quran 4:112

Arabic:

وَمَن يَكْسِبْ خَطِيئَةً أَوْ إِثْمًا ثُمَّ يَرْمِ بِهِ بَرِيئًا فَقَدِ احْتَمَلَ بُهْتَانًا وَإِثْمًا مُّبِينًا

Transliteration: Wa man yaksib khaṭī’atan aw ithman thumma yarmi bihi barī’an faqadi iḥtamala buhtānan wa ithman mubīnā.

Translation: "Whoever commits a sin or error and then blames it on an innocent person has borne a grave slander and a manifest sin."

Context: Reiterates the gravity of false accusations.

________________________________________

Key Lessons

• Integrity in Speech: The Quran repeatedly ties truthfulness to faith (e.g., 16:105).

• Social Justice: Lies harm individuals and communities (2:283, 83:1–3).

• Divine Accountability: Every word and intention is recorded (50:18).

• Rejecting Harm: Silence in the face of falsehood is complicity (24:15–16).

These verses collectively establish a robust ethical framework for combating misinformation, urging believers to prioritize truth, fairness, and responsibility in all communication.

Item number eight states that the phrase

وَمَن يَكْسِبْ خَطِيئَةً أَوْ إِثْمًا ثُمَّ يَرْمِ بِهِ بَرِيئًا فَقَدِ احْتَمَلَ بُهْتَانًا وَإِثْمًا مُّبِينًا

is from Quran 6:144 but it is actually the entire verse Quran 4:112

The correct verse from Quran 6:144 is:

وَمِنَ ٱلۡإِبِلِ ٱثۡنَيۡنِ وَمِنَ ٱلۡبَقَرِ ٱثۡنَيۡنِ‌ۗ قُلۡ ءَآلذَّڪَرَيۡنِ حَرَّمَ أَمِ ٱلۡأُنثَيَيۡنِ أَمَّا ٱشۡتَمَلَتۡ عَلَيۡهِ أَرۡحَامُ ٱلۡأُنثَيَيۡنِ‌ۖ أَمۡ ڪُنتُمۡ شُہَدَآءَ إِذۡ وَصَّٮٰڪُمُ ٱللَّهُ بِهَـٰذَا‌ۚ فَمَنۡ أَظۡلَمُ مِمَّنِ ٱفۡتَرَىٰ عَلَى ٱللَّهِ ڪَذِبً۬ا لِّيُضِلَّ ٱلنَّاسَ بِغَيۡرِ عِلۡمٍ‌ۗ إِنَّ ٱللَّهَ لَا يَہۡدِى ٱلۡقَوۡمَ ٱلظَّـٰلِمِينَ 

The translation is:

And of the camels two (male and female), and of oxen two (male and female). Say: "Has He forbidden the two males or the two females or (the young) which the wombs of the two females enclose? Or were you present when Allâh ordered you such a thing? Then who does more wrong than one who invents a lie against Allâh, to lead mankind astray without knowledge. Certainly Allâh guides not the people who are Zâlimûn (polytheists and wrong-doers)."

It is interesting that item number nine in the answer, right below number eight, is the same verse but correctly referenced this time.

Wrong chapter and verse number from DeepSeek

I asked DeepSeek for additional verses, and this time, DeepSeek started to hallucinate. The answer included a phrase that does not exist in the Quran at all, but DeepSeek attributed the phrase to a specific verse in the Quran.

Me: Give me additional verses

Thought for 37 seconds

DeepSeek: Okay, the user has asked for additional Quranic verses related to misinformation again. Let me see. They previously requested verses in Arabic with translations, and I provided several. Now they want more.

First, I need to recall which verses I've already listed. The previous ones included 2:42, 49:6, 24:15-16, 17:36, 50:18, 25:72, 49:12, and then in the last response, I added 17:35, 5:8, 23:3, 22:30, 2:283, 16:105, 83:1-3, 6:144, 4:112. So I need to find more verses that haven't been mentioned yet but still address truthfulness, falsehood, gossip, or accountability in speech.

Let me check other relevant verses. Surah Al-Ma'idah 5:2 comes to mind about cooperation in righteousness. Also, Surah Al-Isra 17:34 about fulfilling promises. Surah Al-Hujurat 49:11 prohibits insulting others. Surah Al-Mutaffifin 83:29-36 talks about the consequences for those who mock believers. Surah Al-Baqarah 2:188 warns against deceit in property. Surah Al-Ahzab 33:70-71 encourages truthful speech. Surah Al-Mumtahanah 60:12 on not hiding testimony. Surah Al-Nahl 16:116 against lying about what's permissible. And Surah Al-Tawbah 9:119 urges being with the truthful.

I need to present these in the same format as before: Arabic, transliteration, translation, and context. Make sure the translations are accurate and the contexts clearly relate to misinformation or ethical communication. Also, check if any of these were already covered. For example, 2:283 was mentioned before, so I should avoid repetition. Wait, 2:188 is about property, which is a form of deceit, so it's relevant. Similarly, Surah Al-Tawbah 9:119 is about being with the truthful, which supports truthfulness in general.

Ensure that each verse adds value to the topic. For instance, Surah Al-Ma'idah 5:2 encourages cooperation in good deeds and avoiding sin, which ties into promoting truth and avoiding falsehood. Surah Al-Isra 17:34 about fulfilling promises relates to honesty. Surah Al-Hujurat 49:11 is about not insulting, which is a form of harmful speech.

Double-check the translations for accuracy. Maybe use Sahih International or another reputable translation. Also, verify the Arabic script to ensure it's correct. Organize them in a logical order, perhaps by theme. Start with cooperation in righteousness, then fulfilling promises, avoiding insults, mocking others, deceit in transactions, truthful speech, hiding testimony, lying about halal/haram, and the call to be with the truthful.

Finally, summarize the key lessons from these additional verses to reinforce the Quran's stance on truthfulness and against misinformation. Make sure the response is clear and helpful, providing the user with meaningful content that expands on the previous answers without redundancy.

Certainly! Here are additional Quranic verses in Arabic that emphasize truthfulness, condemn deception, and guide ethical communication, along with translations and contexts:

________________________________________

1. Cooperation in Righteousness, Not Sin

Quran 5:2

Arabic:

وَتَعَاوَنُوا عَلَى الْبِرِّ وَالتَّقْوَىٰ ۖ وَلَا تَعَاوَنُوا عَلَى الْإِثْمِ وَالْعُدْوَانِ

Transliteration: Wa taʿāwanū ʿalā al-birri wa al-taqwā wa lā taʿāwanū ʿalā al-ithmi wa al-ʿudwān.

Translation: "Cooperate in righteousness and piety, and do not cooperate in sin and aggression."

Context: Encourages collective efforts to uphold truth and justice while rejecting complicity in falsehood or harm.

________________________________________

2. Fulfilling Promises and Trusts

Quran 17:34

Arabic:

وَأَوْفُوا بِالْعَهْدِ ۖ إِنَّ الْعَهْدَ كَانَ مَسْئُولًا

Transliteration: Wa awfū bil-ʿahdi inna al-ʿahda kāna mas’ūlā.

Translation: "Fulfill your promises, for you will surely be held accountable for them."

Context: Highlights the importance of honesty in commitments, a form of truthful communication.

________________________________________

3. Prohibition of Insulting Others

Quran 49:11

Arabic:

وَلَا تَلْمِزُوا أَنفُسَكُمْ وَلَا تَنَابَزُوا بِالْأَلْقَابِ

Transliteration: Wa lā talmizū anfusakum wa lā tanāabazū bil-alqāb.

Translation: "Do not insult one another or call each other by offensive nicknames."

Context: Forbids demeaning speech, which can fuel division and false narratives.

________________________________________

4. Condemnation of Mockery

Quran 83:29–36

Arabic:

إِنَّ الَّذِينَ أَجْرَمُوا كَانُوا مِنَ الَّذِينَ آمَنُوا يَضْحَكُونَ. وَإِذَا مَرُّوا بِهِمْ يَتَغَامَزُونَ

Transliteration: Inna alladhīna ajramū kānū mina alladhīna āmanū yaḍḥakūn. Wa idhā marrū bihim yataghāmazūn.

Translation: "Indeed, the wicked used to laugh at the believers, winking [mockingly] when they passed by them."

Context: Condemns mockery and belittling others, behaviors that often accompany misinformation.

________________________________________

5. Prohibition of Deceit in Transactions

Quran 2:188

Arabic:

وَلَا تَأْكُلُوا أَمْوَالَكُم بَيْنَكُم بِالْبَاطِلِ

Transliteration: Wa lā ta’kulū amwālakum baynakum bil-bāṭil.

Translation: "Do not consume one another’s wealth unjustly."

Context: Forbids deceitful practices, including lying for financial gain.

________________________________________

6. Command to Speak Truthfully

Quran 33:70–71

Arabic:

يَا أَيُّهَا الَّذِينَ آمَنُوا اتَّقُوا اللَّهَ وَقُولُوا قَوْلًا سَدِيدًا. يُصْلِحْ لَكُمْ أَعْمَالَكُمْ

Transliteration: Yā ayyuhalladhīna āmanū ittaqū Allāha wa qūlū qawlan sadīdā. Yuṣliḥ lakum aʿmālakum.

Translation: "O believers! Fear Allah and speak uprightly. He will mend your deeds for you."

Context: Links truthful speech to divine guidance and rectification of actions.

________________________________________

7. Rejecting Hidden Testimony

Quran 60:12

Arabic:

يَا أَيُّهَا النَّبِيُّ إِذَا جَاءَكَ الْمُؤْمِنَاتُ يُبَايِعْنَكَ عَلَىٰ أَن لَّا يُشْرِكْنَ بِاللَّهِ... وَلَا يَكْتُمْنَ شَهَادَةً

Transliteration: Yā ayyuha al-nabiyyu idhā jā’aka al-mu’minātu yubāyiʿnaka ʿalā an lā yush’rikna billāh... Wa lā yaktumna shahādah.

Translation: "O Prophet! When believing women pledge allegiance to you... [they must] not conceal testimony."

Context: Requires transparency in testimony, rejecting the suppression of truth.

________________________________________

8. Condemnation of False Oaths

Quran 16:116

Arabic:

وَلَا تَقُولُوا لِمَا تَصِفُ أَلْسِنَتُكُمُ الْكَذِبَ هَٰذَا حَلَالٌ وَهَٰذَا حَرَامٌ

Transliteration: Wa lā taqūlū limā taṣifu alsinatukumu al-kaḏiba hādhā ḥalālun wa hādhā ḥarām.

Translation: "Do not say about what your tongues falsely claim, ‘This is lawful, and this is forbidden.’"

Context: Condemns lying to legitimize or prohibit matters falsely.

________________________________________

9. The Call to Be with the Truthful

Quran 9:119

Arabic:

يَا أَيُّهَا الَّذِينَ آمَنُوا اتَّقُوا اللَّهَ وَكُونُوا مَعَ الصَّادِقِينَ

Transliteration: Yā ayyuhalladhīna āmanū ittaqū Allāha wa kūnū maʿa al-ṣādiqīn.

Translation: "O believers! Fear Allah and be with those who are true."

Context: Urges alignment with truthful people, rejecting circles of falsehood.

________________________________________

Key Lessons from These Verses

1. Collective Responsibility: Uphold truth collectively (5:2) and avoid complicity in lies.

2. Integrity in Transactions: Deceit for gain is forbidden (2:188).

3. Dignity in Speech: Mockery, insults, and belittling others are condemned (49:11, 83:29–36).

4. Transparency: Concealing testimony or distorting facts violates trust (60:12, 16:116).

5. Divine Alignment: Truthfulness aligns believers with God’s guidance (33:70–71).

These verses reinforce the Quran’s comprehensive framework for combating misinformation, emphasizing that truth is a divine mandate and lies are a moral and spiritual corruption. In an age of misinformation, these teachings call for vigilance, integrity, and collective accountability.

Item number seven in the answer attributed the phrase:

يَا أَيُّهَا النَّبِيُّ إِذَا جَاءَكَ الْمُؤْمِنَاتُ يُبَايِعْنَكَ عَلَىٰ أَن لَّا يُشْرِكْنَ بِاللَّهِ... وَلَا يَكْتُمْنَ شَهَادَةً

to Quran 60:12

The first sentence exists in Quran 60:12 but the second sentence, after the three dots, does not exist in the Quran at all. It translates to "And them (believing women) will not conceal a testimony".

The correct Quran 60:12 is:

يَـٰٓأَيُّہَا ٱلنَّبِىُّ إِذَا جَآءَكَ ٱلۡمُؤۡمِنَـٰتُ يُبَايِعۡنَكَ عَلَىٰٓ أَن لَّا يُشۡرِكۡنَ بِٱللَّهِ شَيۡـًٔ۬ا وَلَا يَسۡرِقۡنَ وَلَا يَزۡنِينَ وَلَا يَقۡتُلۡنَ أَوۡلَـٰدَهُنَّ وَلَا يَأۡتِينَ بِبُهۡتَـٰنٍ۬ يَفۡتَرِينَهُ ۥ بَيۡنَ أَيۡدِيہِنَّ وَأَرۡجُلِهِنَّ وَلَا يَعۡصِينَكَ فِى مَعۡرُوفٍ۬‌ۙ فَبَايِعۡهُنَّ وَٱسۡتَغۡفِرۡ لَهُنَّ ٱللَّهَ‌ۖ إِنَّ ٱللَّهَ غَفُورٌ۬ رَّحِيمٌ۬ 

The translation is:

O Prophet! When believing women come to you to give you the Bai'âh (pledge), that they will not associate anything in worship with Allah, that they will not steal, that they will not commit illegal sexual intercourse, that they will not kill their children, that they will not utter slander, intentionally forging falsehood (i.e. by making illegal children belonging to their husbands), and that they will not disobey you in Ma'rûf (Islâmic Monotheism and all that which Islâm ordains) then accept their Bai'âh (pledge), and ask Allâh to forgive them, Verily, Allâh is Oft-Forgiving, Most Merciful.

Wrong answer from DeepSeek

I stopped asking DeepSeek at this point because after performing this experiment on Google Gemini, ChatGPT, and now DeepSeek and getting the same type of errors and hallucinations, I am convinced that LLMs are not capable of correctly quoting verses from the Quran in Arabic. In addition, archiving my questions and DeepSeek's answers by public web archives is not possible because the chat page is behind a login wall which makes it not archivable.

Conclusions

I initially was looking to study LLMs' improvements over the past two years quoting verses from the Quran. Although testing the same prompts in the paper did not produce any errors, slightly changing the prompts while keeping the same meaning resulted in producing errors and hallucinations in the answers using all three tools (Google Gemini, ChatGPT, and DeepSeek). Some researchers warned that misinformation and disinformation cannot be eliminated from LLMs output. They argue that LLMs' errors will continue to increase and result in a total collapse of the model.

The following table outlines the errors each model produced and the experiment year:

Types of errors produced by Google Gemini, ChatGPT, and DeepSeek

Retrieval-Augmented Generation (RAG) might be a promising technique that can significantly reduce errors and hallucinations in LLMs, especially for tasks that involve citing Quranic verses since the Quran has a fixed structure of 114 chapters (Surahs) and a set number of verses (Ayahs). This textual stability makes it an ideal candidate for RAG. The model doesn't need to "guess" or "infer" the content of a verse; it can retrieve the exact text. Therefore, for citing Quranic verses, it is possible that RAG LLM can significantly enhance the reliability and trustworthiness of AI-generated responses involving Quranic citations.

--Hussam Hallak