2025-05-06: Part 2 - Large Language Models (LLMs) are hallucinating in Arabic about the Quran (Open AI ChatGPT-4o)
Introduction
I discussed LLMs' hallucinations and demonstrated their inability to correctly cite Quranic verses using Google Gemini as an example in part 1 of this blog post. Google Gemini's answers to different prompts included incorrect Quranic citations and phrases that do not exist in the Quran attributed to specific verses in the Quran. I also explained the importance of correctly citing and presenting Quranic verses in Arabic. Here, in part 2, I present a comparison between Google Gemini and Open AI's ChatGPT. Using the same prompts, ChatGPT-4o hallucinations produced similar results to that of Google Gemini. Again, misinformation remains one of the biggest challenges that compromises LLMs' credibility quoting verses from the Quran in Arabic. Experts in Classical Arabic and those who memorize the Quran word for word can identify errors in ChatGPT's output, however, less educated people will have difficulty distinguishing between correct and incorrect answers from ChatGPT. Some Arabic speakers have examined ChatGPT's answers and found them to have significant errors when referencing the Quran.
Improvements in ChatGPT over the last two years
The paper I reviewed in part 1 of this post tested ChatGPT-3.5 in 2023 using a sample of prompts regarding the Quran's views on misinformation. The authors found the output to have significant and consistent errors. They regenerated queries to the question “What does the Quran have to say about misinformation?”. They also sampled responses from ChatGPT with the additional prompt “please cite verses in Arabic”. The errors in the output ranged from producing correctly quoted and cited, yet irrelevant Quranic material to misordering the words of Quranic verses; the errors were not minor. They also found hallucinations in ChatGPT's answers attributing text that doesn't exist in the Quran at all to specific chapters and verses in the Quran. I repeated their experiment on ChatGPT-4o in January 2025 and I was unable to reproduce their output. I did not get incorrect Quranic citations or verses that did not exist in the Quran. I did not get any altered verses from the Quran in any way. Every output was correct, accurate, and authentic.
 |
| Correct answers from ChatGPT |
However, when I repeated the experiment I did with Google Gemini in part 1 of this post in January of 2025 and asked ChatGPT for additional verses in Arabic, the results were not much better than the ones I got from Google Gemini. ChatGPT provided the wrong reference one time when I asked for additional verses on misinformation. In addition, ChatGPT provided phrases that do not exist in the Quran and attributed them to specific verses in the Quran when I tried different prompts that are not in the paper, but they have the same meaning. Although the errors in ChatGPT's output are not as bad as the ones I found in Google Gemini's, I think the impact is worse because the wrong verses are closer to correct verses in the Quran, and they can definitely be taken as correct verses to the non-expert. The difference was in one letter in one case and one word in another case. The output's syntax was also wrong in the latter.
 |
| Wrong chapter and verses' numbers from ChatGPT |
Me: Give me additional verses
ChatGPT:
Certainly! The Quran repeatedly emphasizes the dangers of misinformation, falsehoods, and irresponsible speech. Here are additional verses in Arabic that address this issue:
6. Do Not Spread Falsehood Knowingly
📖 Surah Aal-e-Imran (3:78)
وَإِنَّ مِنْهُمْ لَفَرِيقًا يَلْوُونَ أَلْسِنَتَهُمْ بِالْكِتَابِ لِتَحْسَبُوهُ مِنَ الْكِتَابِ وَمَا هُوَ مِنَ الْكِتَابِ وَيَقُولُونَ هُوَ مِنْ عِنْدِ اللَّهِ وَمَا هُوَ مِنْ عِنْدِ اللَّهِ وَيَقُولُونَ عَلَى اللَّهِ الْكَذِبَ وَهُمْ يَعْلَمُونَ
🔹 "And indeed, there is among them a party who alter the Scripture with their tongues so that you may think it is from the Scripture, but it is not from the Scripture. And they say, 'This is from Allah,' but it is not from Allah. And they speak a lie against Allah while they know [it]."
7. Lying Brings a Curse from Allah
📖 Surah Aal-e-Imran (3:61)
فَمَنْ حَاجَّكَ فِيهِ مِنْ بَعْدِ مَا جَاءَكَ مِنَ الْعِلْمِ فَقُلْ تَعَالَوْا نَدْعُ أَبْنَاءَنَا وَأَبْنَاءَكُمْ وَنِسَاءَنَا وَنِسَاءَكُمْ وَأَنْفُسَنَا وَأَنْفُسَكُمْ ثُمَّ نَبْتَهِلْ فَنَجْعَلْ لَعْنَتَ اللَّهِ عَلَى الْكَاذِبِينَ
🔹 "Then whoever argues with you about it after knowledge has come to you, say, 'Come, let us call our sons and your sons, our women and your women, ourselves and yourselves, then supplicate earnestly and invoke the curse of Allah upon the liars.'"
8. Do Not Falsify Facts or Distort the Truth
📖 Surah Al-Ma’idah (5:41)
يُحَرِّفُونَ الْكَلِمَ مِنْ بَعْدِ مَوَاضِعِهِ
🔹 "They distort words from their [proper] places..."
9. Lying is from Shaytan (Satan)
📖 Surah Al-Mujadilah (58:14-15)
وَإِذَا رَأَيْتَهُمْ تُعْجِبُكَ أَجْسَامُهُمْ وَإِنْ يَقُولُوا تَسْمَعْ لِقَوْلِهِمْ كَأَنَّهُمْ خُشُبٌ مُسَنَّدَةٌ ۖ يَحْسَبُونَ كُلَّ صَيْحَةٍ عَلَيْهِمْ ۚ هُمُ الْعَدُوُّ فَاحْذَرْهُمْ ۚ قَاتَلَهُمُ اللَّهُ أَنَّىٰ يُؤْفَكُونَ
🔹 "And when you see them, their forms please you, and if they speak, you listen to their speech. [They are] like pieces of wood propped up – they think that every shout is against them. They are the enemy, so beware of them. May Allah destroy them; how are they deluded?"
10. Lying Leads to the Fire
📖 Surah Az-Zumar (39:60)
وَيَوْمَ الْقِيَامَةِ تَرَى الَّذِينَ كَذَبُوا عَلَى اللَّهِ وُجُوهُهُمْ مُسْوَدَّةٌ ۚ أَلَيْسَ فِي جَهَنَّمَ مَثْوًى لِلْمُتَكَبِّرِينَ
🔹 "And on the Day of Resurrection, you will see those who lied against Allah – their faces will be blackened. Is there not in Hell a residence for the arrogant?"
11. Truthfulness Leads to Salvation
📖 Surah Al-Ahzab (33:70-71)
يَا أَيُّهَا الَّذِينَ آمَنُوا اتَّقُوا اللَّهَ وَقُولُوا قَوْلًا سَدِيدًا يُصْلِحْ لَكُمْ أَعْمَالَكُمْ وَيَغْفِرْ لَكُمْ ذُنُوبَكُمْ ۗ وَمَنْ يُطِعِ اللَّهَ وَرَسُولَهُ فَقَدْ فَازَ فَوْزًا عَظِيمًا
🔹 "O you who have believed, fear Allah and speak words of appropriate justice. He will [then] amend for you your deeds and forgive you your sins. And whoever obeys Allah and His Messenger has certainly attained a great attainment."
Conclusion
The Quran strongly condemns misinformation, distortion, and falsehood. It teaches that:
✅ Truthfulness leads to success
❌ Spreading falsehoods leads to regret and punishment
⚠️ Lies cause harm, injustice, and corruption in society
Would you like a deeper explanation of any verse? 😊
The 9th quoted verse (highlighted in red) is a verse from the Quran, but it is not in Quran 58:14-15. It is in Quran 63:4. The two verses in Quran 58:14-15 are:
أَلَمْ تَرَ إِلَى الَّذِينَ تَوَلَّوْا قَوْمًا غَضِبَ اللَّهُ عَلَيْهِم مَّا هُم مِّنكُمْ وَلَا مِنْهُمْ وَيَحْلِفُونَ عَلَى الْكَذِبِ وَهُمْ يَعْلَمُونَ (14) أَعَدَّ اللَّهُ لَهُمْ عَذَابًا شَدِيدًا ۖ إِنَّهُمْ سَاءَ مَا كَانُوا يَعْمَلُونَ (15)
The translation is:
Have you noted those who befriended people with whom GOD is angry? They neither belong with you, nor with them. They deliberately swear lies! (14) GOD has prepared for them a severe retribution. Miserable indeed is what they used to do. (15)
ChatGPT correctly cited the Quran 63:4 (highlighted green) in the following answer when I repeated the prompt.
Me: Give me additional verses
ChatGPT:
Certainly! The Quran contains many verses warning against misinformation, lying, and spreading falsehoods. Here are additional verses in Arabic, along with their meanings:
12. Lying is a Major Sin
📖 Surah Az-Zumar (39:32)
فَمَنْ أَظْلَمُ مِمَّنْ كَذَبَ عَلَى اللَّهِ وَكَذَّبَ بِالصِّدْقِ إِذْ جَاءَهُ ۚ أَلَيْسَ فِي جَهَنَّمَ مَثْوًى لِلْكَافِرِينَ
🔹 "So who is more unjust than one who lies about Allah and denies the truth when it has come to him? Is there not in Hell a residence for the disbelievers?"
13. Speaking Falsehood is a Sign of Hypocrisy
📖 Surah At-Tawbah (9:77)
فَأَعْقَبَهُمْ نِفَاقًا فِي قُلُوبِهِمْ إِلَىٰ يَوْمِ يَلْقَوْنَهُ بِمَا أَخْلَفُوا اللَّهَ مَا وَعَدُوهُ وَبِمَا كَانُوا يَكْذِبُونَ
🔹 "So He penalized them with hypocrisy in their hearts until the day they will meet Him – because they failed Allah in what they promised Him and because they used to lie."
14. Allah Hates Falsehood
📖 Surah An-Nahl (16:105)
إِنَّمَا يَفْتَرِي الْكَذِبَ الَّذِينَ لَا يُؤْمِنُونَ بِآيَاتِ اللَّهِ وَأُو۟لَٰئِكَ هُمُ ٱلْكَٰذِبُونَ
🔹 "Only those who do not believe in the verses of Allah fabricate falsehood, and it is they who are the liars."
15. Spreading Falsehood Corrupts Society
📖 Surah Al-Baqarah (2:191)
وَالْفِتْنَةُ أَشَدُّ مِنَ الْقَتْلِ
🔹 "And fitnah (spreading corruption and falsehood) is worse than killing."
16. Lying Leads to Destruction
📖 Surah Al-Ankabut (29:3)
وَلَقَدْ فَتَنَّا الَّذِينَ مِنْ قَبْلِهِمْ ۖ فَلَيَعْلَمَنَّ اللَّهُ الَّذِينَ صَدَقُوا وَلَيَعْلَمَنَّ الْكَاذِبِينَ
🔹 "And We have certainly tested those before them, and Allah will surely make evident those who are truthful, and He will surely make evident the liars."
17. Spreading False News Causes Harm
📖 Surah Al-Ahzab (33:58)
وَالَّذِينَ يُؤْذُونَ الْمُؤْمِنِينَ وَالْمُؤْمِنَاتِ بِغَيْرِ مَا اكْتَسَبُوا فَقَدِ احْتَمَلُوا بُهْتَانًا وَإِثْمًا مُبِينًا
🔹 "And those who harm believing men and believing women for [something] other than what they have earned have certainly borne upon themselves a slander and manifest sin."
18. False Speech Leads to Hellfire
📖 Surah Al-Mutaffifin (83:10-12)
وَيْلٌ يَوْمَئِذٍ لِلْمُكَذِّبِينَ الَّذِينَ يُكَذِّبُونَ بِيَوْمِ الدِّينِ وَمَا يُكَذِّبُ بِهِ إِلَّا كُلُّ مُعْتَدٍ أَثِيمٍ
🔹 "Woe that Day to the deniers, those who deny the Day of Recompense. And none denies it except every sinful transgressor."
19. Hypocrites Spread Falsehood
📖 Surah Al-Munafiqun (63:4)
وَإِذَا رَأَيْتَهُمْ تُعْجِبُكَ أَجْسَامُهُمْ ۖ وَإِنْ يَقُولُوا تَسْمَعْ لِقَوْلِهِمْ ۖ كَأَنَّهُمْ خُشُبٌ مُسَنَّدَةٌ ۖ يَحْسَبُونَ كُلَّ صَيْحَةٍ عَلَيْهِمْ ۚ هُمُ الْعَدُوُّ فَاحْذَرْهُمْ ۚ قَاتَلَهُمُ اللَّهُ ۖ أَنَّىٰ يُؤْفَكُونَ
🔹 "And when you see them, their forms please you, and if they speak, you listen to their speech. [They are] like pieces of wood propped up – they think that every shout is against them. They are the enemy, so beware of them. May Allah destroy them; how are they deluded?"
20. Allah Commands Speaking the Truth
📖 Surah Al-An’am (6:152)
وَإِذَا قُلْتُمْ فَاعْدِلُوا وَلَوْ كَانَ ذَا قُرْبَىٰ
🔹 "And when you speak, be just, even if it concerns a near relative."
Conclusion
The Quran provides a strong warning against spreading misinformation, lying, and distorting the truth. It encourages:
✅ Speaking truthfully and justly
✅ Verifying information before spreading it
✅ Avoiding false accusations and rumors
Would you like more explanations or interpretations of any of these verses? 😊
I tried the same prompts that made Google Gemini hallucinate; the output to the first prompt was correct.
 |
| Correct answers from ChatGPT |
Me: Give me Quran verses in Arabic about falsehood
ChatGPT:
Certainly! Here are several Quranic verses in Arabic about falsehood (الباطل والكذب):
1. Falsehood Will Perish
📖 Surah Al-Isra (17:81)
وَقُلْ جَاءَ الْحَقُّ وَزَهَقَ الْبَاطِلُ ۚ إِنَّ الْبَاطِلَ كَانَ زَهُوقًا
🔹 "And say: Truth has come, and falsehood has vanished. Indeed, falsehood is bound to perish."
2. Lying is a Great Sin
📖 Surah Az-Zumar (39:3)
أَلَا لِلَّهِ الدِّينُ الْخَالِصُ ۚ وَالَّذِينَ اتَّخَذُوا مِن دُونِهِ أَوْلِيَاءَ مَا نَعْبُدُهُمْ إِلَّا لِيُقَرِّبُونَا إِلَى اللَّهِ زُلْفَىٰ ۚ إِنَّ اللَّهَ يَحْكُمُ بَيْنَهُمْ فِي مَا هُمْ فِيهِ يَخْتَلِفُونَ ۗ إِنَّ اللَّهَ لَا يَهْدِي مَنْ هُوَ كَاذِبٌ كَفَّارٌ
🔹 "Unquestionably, for Allah is the pure religion. And those who take protectors besides Him [say], 'We only worship them that they may bring us nearer to Allah in position.' Indeed, Allah will judge between them concerning that over which they differ. Indeed, Allah does not guide one who is a liar and disbeliever."
3. Falsehood Leads to Hellfire
📖 Surah Al-Mutaffifin (83:10-12)
وَيْلٌ يَوْمَئِذٍ لِلْمُكَذِّبِينَ الَّذِينَ يُكَذِّبُونَ بِيَوْمِ الدِّينِ وَمَا يُكَذِّبُ بِهِ إِلَّا كُلُّ مُعْتَدٍ أَثِيمٍ
🔹 "Woe that Day to the deniers, those who deny the Day of Recompense. And none denies it except every sinful transgressor."
4. Falsehood Brings Allah’s Curse
📖 Surah An-Nahl (16:105)
إِنَّمَا يَفْتَرِي الْكَذِبَ الَّذِينَ لَا يُؤْمِنُونَ بِآيَاتِ اللَّهِ وَأُو۟لَٰئِكَ هُمُ ٱلْكَٰذِبُونَ
🔹 "Only those who do not believe in the verses of Allah fabricate falsehood, and it is they who are the liars."
5. Do Not Conceal the Truth
📖 Surah Al-Baqarah (2:42)
وَلَا تَلْبِسُوا الْحَقَّ بِالْبَاطِلِ وَتَكْتُمُوا الْحَقَّ وَأَنْتُمْ تَعْلَمُونَ
🔹 "And do not mix the truth with falsehood or conceal the truth while you know [it]."
6. False Speech is Forbidden
📖 Surah Al-Hajj (22:30)
فَاجْتَنِبُوا الرِّجْسَ مِنَ الْأَوْثَانِ وَاجْتَنِبُوا قَوْلَ الزُّورِ
🔹 "So avoid the impurity of idols and avoid false speech."
7. The Wicked Spread Falsehood
📖 Surah Al-Jathiyah (45:7)
وَيْلٌ لِكُلِّ أَفَّاكٍ أَثِيمٍ
🔹 "Woe to every sinful liar."
8. Liars Will Be Held Accountable
📖 Surah Qaf (50:18)
مَّا يَلْفِظُ مِن قَوْلٍ إِلَّا لَدَيْهِ رَقِيبٌ عَتِيدٌ
🔹 "Man does not utter any word except that with him is an observer prepared [to record]."
9. Do Not Follow Falsehood
📖 Surah Al-Furqan (25:72)
وَالَّذِينَ لَا يَشْهَدُونَ الزُّورَ وَإِذَا مَرُّوا بِاللَّغْوِ مَرُّوا كِرَامًا
🔹 "And [the servants of the Most Merciful are] those who do not testify to falsehood, and when they pass near ill speech, they pass by with dignity."
10. Liars Will Be Disgraced
📖 Surah Al-Kahf (18:5)
مَا لَهُم بِهِۦ مِنْ عِلْمٍ وَلَا لِأَبَآئِهِمْ ۚ كَبُرَتْ كَلِمَةً تَخْرُجُ مِنْ أَفْوَٰهِهِمْ ۚ إِن يَقُولُونَ إِلَّا كَذِبًا
🔹 "They have no knowledge of it, nor do their forefathers. Grave is the word that comes out of their mouths; they speak not except a lie."
Conclusion
The Quran strongly condemns falsehood, lies, and deception. It teaches that:
✅ Truthfulness is a virtue
❌ Falsehood leads to loss and punishment
⚠️ Spreading lies and misinformation is a major sin
Would you like more verses or interpretations? 😊
I asked for additional verses and that's when worse errors began to take place.
 |
| Wrong answers from ChatGPT |
 |
| Wrong answers from ChatGPT |
Me: Give me more verses
ChatGPT:
Certainly! Here are more Quranic verses in Arabic that address falsehood (الباطل) and lying (الكذب):
11. The Falsehood of Idol Worship
📖 Surah Al-Ankabut (29:17)
إِنَّمَا تَعْبُدُونَ مِن دُونِ اللَّهِ أَوْثَٰنًا وَتَخْلُقُونَ إِفْكًا ۚ إِنَّ الَّذِينَ تَعْبُدُونَ مِن دُونِ اللَّهِ لَا يَمْلِكُونَ لَكُمْ رِزْقًا فَٱبْتَغُوا۟ عِندَ ٱللَّهِ ٱلرِّزْقَ وَٱعْبُدُوهُ وَٱشْكُرُوا۟ لَهُۥٓ ۖ إِلَيْهِ تُرْجَعُونَ
🔹 "You only worship, besides Allah, idols, and you invent falsehood. Indeed, those you worship besides Allah do not possess [for] you [the ability to provide] sustenance. So seek sustenance from Allah and worship Him and be grateful to Him. To Him you will be returned."
12. Liars Will Be Exposed on the Day of Judgment
📖 Surah Az-Zumar (39:60)
وَيَوْمَ ٱلْقِيَٰمَةِ تَرَى ٱلَّذِينَ كَذَبُوا۟ عَلَى ٱللَّهِ وُجُوهُهُم مُّسْوَدَّةٌ ۚ أَلَيْسَ فِى جَهَنَّمَ مَثْوًىۭ لِّلْمُتَكَبِّرِينَ
🔹 "And on the Day of Resurrection, you will see those who lied about Allah with their faces blackened. Is there not in Hell a residence for the arrogant?"
13. Falsehood is a Tool of the Corrupt
📖 Surah Al-Baqarah (2:78-79)
وَمِنْهُمْ أُمِّيُّونَ لَا يَعْلَمُونَ الْكِتَابَ إِلَّا أَمَانِيَّ وَإِنْ هُمْ إِلَّا يَظُنُّونَ فَوَيْلٌ لِلَّذِينَ يَكْتُبُونَ الْكِتَابَ بِأَيْدِيهِمْ ثُمَّ يَقُولُونَ هَٰذَا مِنْ عِندِ اللَّهِ لِيَشْتَرُوا بِهِ ثَمَنًا قَلِيلًا ۖ فَوَيْلٌ لَهُمْ مِمَّا كَتَبَتْ أَيْدِيهِمْ وَوَيْلٌ لَهُمْ مِمَّا يَكْسِبُونَ
🔹 "And among them are unlettered ones who do not know the Scripture except [indulgement in] wishful thinking, but they are only assuming. So woe to those who write the 'scripture' with their own hands, then say, 'This is from Allah,' in order to exchange it for a small price. Woe to them for what their hands have written, and woe to them for what they earn."
14. Falsehood Leads to Corruption
📖 Surah Al-Ma’idah (5:41)
يَسْمَعُونَ لِلْكَذِبِ أَكَّالُونَ لِلسُّحْتِ
🔹 "They listen to falsehood and consume forbidden gain..."
15. False Oaths Do Not Protect Liars
📖 Surah An-Nahl (16:94)
وَلَا تَتَّخِذُوا أَيْمَانَكُمْ دَخَلًا بَيْنَكُمْ فَتَزِلَّ قَدَمٌ بَعْدَ ثُبُوتِهَا وَتَذُوقُوا السُّوءَ بِمَا صَدَدتُّمْ عَن سَبِيلِ اللَّهِ وَلَكُمْ عَذَابٌ عَظِيمٌ
🔹 "And do not take your oaths as a means of deception among yourselves, lest a foot slip after it was firmly planted, and you would taste evil for what you diverted from the way of Allah, and you would have a great punishment."
16. Satan Uses Falsehood to Mislead
📖 Surah Al-A’raf (7:27)
يَٰبَنِىٓ ءَادَمَ لَا يَفْتِنَنَّكُمُ ٱلشَّيْطَٰنُ كَمَآ أَخْرَجَ أَبَوَيْكُم مِّنَ ٱلْجَنَّةِ يَنزِعُ عَنْهُمَا لِبَاسَهُمَا لِيُرِيَهُمَا سَوْءَٰتِهِمَا ۚ إِنَّهُۥ يَرَىٰكُمْ هُوَ وَقَبِيلُهُۥ مِنْ حَيْثُ لَا تَرَوْنَهُمْ ۗ إِنَّا جَعَلْنَا ٱلشَّيَٰطِينَ أَوْلِيَٓاءَ لِلَّذِينَ لَا يُؤْمِنُونَ
🔹 "O children of Adam, let not Satan tempt you as he removed your parents from Paradise, stripping them of their clothing to show them their private parts. Indeed, he sees you, he and his tribe, from where you do not see them. Indeed, We have made the devils allies to those who do not believe."
17. Falsehood Brings Eternal Regret
📖 Surah Al-Kahf (18:103-104)
قُلْ هَلْ نُنَبِّئُكُم بِالْأَخْسَرِينَ أَعْمَالًا ٱلَّذِينَ ضَلَّ سَعْيُهُمْ فِى ٱلْحَيَوٰةِ ٱلدُّنْيَا وَهُمْ يَحْسَبُونَ أَنَّهُمْ يُحْسِنُونَ صُنْعًا
🔹 "Say, 'Shall We inform you of the greatest losers in respect to [their] deeds? Those whose effort is lost in worldly life, while they think that they are doing well in work.'"
18. False Promises Are a Tool of Hypocrites
📖 Surah Al-Munafiqun (63:1)
إِذَا جَاءَكَ ٱلْمُنَٰفِقُونَ قَالُوا۟ نَشْهَدُ إِنَّكَ لَرَسُولُ ٱللَّهِ ۗ وَٱللَّهُ يَعْلَمُ إِنَّكَ لَرَسُولُهُۥ وَٱللَّهُ يَشْهَدُ إِنَّ ٱلْمُنَٰفِقِينَ لَكَٰذِبُونَ
🔹 "When the hypocrites come to you, [O Muhammad], they say, 'We testify that you are the Messenger of Allah.' And Allah knows that you are His Messenger, and Allah testifies that the hypocrites are liars."
19. Those Who Lie About Allah Will Face Severe Punishment
📖 Surah Al-Kahf (18:15)
مَنۢ أَظْلَمُ مِمَّنِ ٱفْتَرَىٰ عَلَى ٱللَّهِ كَذِبًا
🔹 "Who is more unjust than one who invents a lie against Allah?"
20. Justice and Truth Will Prevail
📖 Surah Yunus (10:82)
وَيُحِقُّ اللَّهُ الْحَقَّ بِكَلِمَاتِهِ وَلَوْ كَرِهَ الْمُجْرِمُونَ
🔹 "And Allah establishes the truth by His words, even if the criminals dislike it."
Conclusion
The Quran strongly warns against falsehood, deception, and lying. It teaches that:
✅ Truth will always triumph
❌ Falsehood leads to disgrace and punishment
⚠️ Lying is a characteristic of the corrupt and hypocrites
Would you like more explanations or interpretations? 😊
ChatGPT attributed the phrase يَسْمَعُونَ لِلْكَذِبِ أَكَّالُونَ لِلسُّحْتِ to Quran 5:41 in item number 14 in the answer. While this is an incorrect citation and the syntax of the phrase is also incorrect, Quran 5:42 has a very similar phrase سَمَّاعُونَ لِلْكَذِبِ أَكَّالُونَ لِلسُّحْتِ
The phrase in ChatGPT's output translates to: They are listening to lies, devourers of what's forbidden (bribery).
The phrase from Quran 5:42 translates to: Listeners to lies, devourers of what's forbidden (bribery).
I was able to catch this error quickly because I have read the entire Quran multiple times and have seen and heard the correct verse so many times. I absolutely remember it. For someone who haven't seen it or unaware of the wrong syntax, they will not be able to identify this error.
The second error in ChatGPT's output is even worse because the difference is only in one letter. I was only able to identify this error because I checked each and every verse in the output of this entire experiment with an authentic copy of the Quran (word for word). ChatGPT attributed the phrase مَنۢ أَظْلَمُ مِمَّنِ ٱفْتَرَىٰ عَلَى ٱللَّهِ كَذِبًا to Quran 18:15 in item number 19 in the answer. While this is an incorrect citation, the syntax of the phrase is not necessarily incorrect. The only difference between this phrase and a phrase in Quran 18:15 is a deletion of one letter (ف) from the first word in the phrase. The deletion does not change the meaning or make it wrong syntactically since the output is lacking the context (the previous phrase in the same verse). To someone who doesn't memorize this verse word for word, this error cannot be identified.
Quran 18:15 has the phrase فَمَنْ أَظْلَمُ مِمَّنِ افْتَرَىٰ عَلَى اللَّهِ كَذِبًا
The translation of both phrases (the correct one and the wrong one) is:
Who is more evil than the one who fabricates lies and attributes them to GOD?
In January 2025, I was unable to reproduce the errors that Ali-Reza Bhojani and Marcus Schwarting found in 2023 in ChatGPT's output. The output of ChatGPT was consistently correct to the prompts mentioned in the paper. I tried to archive the ChatGPT's answers to the prompts I issued. Although SPN said that a snapshot was captured, when trying to retrieve the archived copy, the page said that the Wayback Machine has not archived that URL.
 |
| The Internet Archive was unable to archive chat with ChatGPT |
It is possible to archive the entire chat(s) with ChatGPT if the user created a public link (shareable) for the chat and submitted it to archive.is
 |
| Creating a public link to chat with ChatGPT |
 |
| The created/updated link from ChatGPT to be shared and archived |
I archived my chat with ChatGPT on archive.is.
Note that this method of archiving the chat with ChatGPT does not work with the Internet Archive. It looks like the page has been saved using SPN, but the page does not replay.
 |
| Using SPN to archive my chat with ChatGPT |
 |
| The Wayback Machine has not archived the link I submitted for archiving |
ChatGPT cannot provide accurate citations from the Quran in Arabic
From this experiment, I found that ChatGPT "improvement" is similar to that of Google Gemini (from part 1 of this post). They both did not produce errors for the same prompts issued in the paper in 2023. The results were all correct citations with no errors, however, the answers had wrong citations to correct verses in the Quran for different prompts that have the same meaning. ChatGPT, as well as Google Gemini, provided phrases that do not exist in the Quran (mixtures of words that exist in the Quran and elsewhere) and attributed them to the Quran. Some of these phrases included syntax errors and altered verses attributing them to the Quran which is problematic for Muslims who believe that the Quran is free from errors and modifications.
After two years and going from ChatGPT-3.5 to ChatGPT-4o, the errors and hallucinations remain present when using slightly different prompts. Some researchers argue that LLMs' errors and hallucinations not only will remain present in all LLMs, but they will increase as more and more LLMs' generated content make it to the training datasets (content on the internet), and result in a total collapse of the model.
Comments
Post a Comment