The Hidden Cost of Trusting AI Without Checking Its Work

AI is good at sounding right. That is exactly why it can be dangerous.

I use AI constantly in development work. It helps me move faster, sketch ideas sooner, and get through routine tasks with less friction. But I do not treat AI output as fact, and I do not think anyone else should either.

There is a big difference between using AI as a starting point and using it as a source of truth. That gap is where the real risk lives.

This is not hypothetical. Courts have sanctioned lawyers for filing briefs with fabricated citations. Hospitals and clinicians have used transcription tools that can invent words the speaker never said. Customer-service chatbots have given false policy information that companies were still held responsible for. Researchers and reviewers are now dealing with AI-generated references that look real enough to slip into published work. ¹ ² ³ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ¹⁰ ¹¹ ¹² ¹³

The common thread is not that AI is useless. It is that AI output becomes dangerous the moment people stop verifying it.

Law Is the Clearest Warning Sign

The legal profession has become one of the clearest examples of what happens when AI-generated output is used without verification.

The best-known case is Mata v. Avianca. A New York attorney submitted a brief containing nonexistent cases generated by ChatGPT. In its sanctions order, the court described the filing as containing bogus judicial decisions along with bogus quotes and bogus internal citations. ¹

That did not turn out to be a one-off embarrassment. By August 2025, one widely cited database tracking AI-related court incidents had logged more than 300 cases globally involving AI-generated hallucinations in litigation and judicial proceedings, including more than 200 recorded in the first eight months of 2025 alone. ²

The consequences have become more serious over time. In July 2025, lawyers representing Mike Lindell were sanctioned after filing a brief that reportedly contained more than two dozen errors, including fabricated citations, and a federal judge fined the attorneys $3,000 each. ³ In California, an appellate court sanctioned attorney Christopher Mostafavi $10,000 in Noland v. Land of the Free for filing briefs with fabricated citations generated by AI. ⁴ Reporting on that ruling also noted that the court criticized opposing counsel for not identifying the fake citations earlier. ¹⁴ In Arizona, a magistrate judge revoked attorney Maren Bam’s pro hac vice status, struck her filing, removed her as counsel, and imposed additional sanctions after fabricated authorities were attributed to real judges. ¹⁵ ¹⁶

None of these incidents happened because the lawyers lacked access to information. They happened because fabricated output was treated as usable without a proper cite check.

That risk is not theoretical. A Stanford-led study found that large language models hallucinated at high rates when asked specific, verifiable legal questions. One version of the paper reported a hallucination range of 69% to 88% for the evaluated models on direct questions about random federal court cases. ¹⁷ ¹⁸

Healthcare Has a Different Failure Mode, but the Same Root Problem

Legal cases are dramatic because sanctions are public. In healthcare, the danger is quieter.

OpenAI’s Whisper has been used in healthcare transcription workflows even though OpenAI’s own model documentation warns against use in “high-risk domains” and in “decision-making contexts, where flaws in accuracy can lead to pronounced flaws in outcomes.” ⁵

Researchers at Cornell found that roughly 1% to 1.4% of tested Whisper transcriptions contained entire hallucinated phrases or sentences that were not present in the underlying audio. The same research found that 38% of the hallucinations included explicit harms such as violence, false personal information, or misleading authority cues. ⁶ ⁷

Reporting on real-world deployments made the problem more concrete. AP reported examples in which one engineer found hallucinations in roughly half of more than 100 hours of transcriptions he reviewed, while another developer reported hallucinations in nearly every one of 26,000 transcripts he analyzed. That reporting also said Whisper-based tooling was being used by more than 30,000 clinicians across 40 health systems or organizations in medical documentation workflows. ⁸

PBS reported that in at least one case involving a Whisper-based workflow, the original audio was discarded after transcription, leaving the AI-generated transcript as the only surviving record of what happened during the appointment. ¹⁹

That is the real issue. The danger is not just that AI can be wrong. It is that people can build processes where the wrong output becomes the official record.

Researchers have also shown that medical chatbots can be pushed into generating dangerous falsehoods with fabricated citations. A 2025 Reuters report on work from Flinders University described leading models that could be induced to produce misinformation, such as false claims about sunscreen and skin cancer, while citing journals that did not actually support the claim. ²⁰

Academia Has a Compounding Problem

In academic work, AI errors can become self-reinforcing.

A fabricated citation is not always obviously fake. AI-generated references often use plausible author names, realistic journal titles, convincing formatting, and DOI-like identifiers, making them harder to spot at a glance. ¹² ¹³

That is where things get worse. Once a fake or corrupted citation slips into a paper, it can be copied into other papers, presentations, bibliographies, or literature reviews. The problem stops being a single hallucination and starts becoming contamination. ¹² ¹³

A 2025 JMIR Mental Health study found that 19.9% of citations generated in its test set were completely fabricated. The same study found that many of the remaining citations still contained bibliographic errors, and its authors concluded that citation fabrication and bibliographic inaccuracies remained common in GPT-4o outputs. ¹¹ ²¹

Post-publication analysis has also raised concerns about peer review catching these errors. GPTZero reported at least 100 confirmed hallucinated citations in accepted NeurIPS 2025 papers, and a later arXiv analysis described 100 fabricated citations appearing across 53 published NeurIPS 2025 papers. ¹² ¹³

The exact numbers will vary by model, prompt, and field. The broader point does not. If a citation looks real enough to pass a quick visual check, then “looks legitimate” is no longer a meaningful standard.

Customer Service Already Shows the Liability Problem

One of the clearest business examples came from Air Canada.

In 2024, a British Columbia tribunal found Air Canada responsible for misinformation from its chatbot about bereavement fares. The user had been told he could seek a refund after booking under conditions that did not match the airline’s actual policy. Air Canada argued, among other things, that the chatbot should be treated as a separate legal entity responsible for its own statements. The tribunal rejected that argument and held the airline responsible for the misinformation on its website. ⁹ ¹⁰

The award totaled C$812.02, consisting of the fare difference, interest, and tribunal fees. The damages were modest, but the lesson was not. A company can still be held responsible when an automated system gives customers false information in the course of business. ⁹ ¹⁰

That matters for any team shipping AI into support, onboarding, quoting, benefits, scheduling, or policy communication.

Defamation Makes the Same Point in a Harder Way

The reputational risk is not limited to internal mistakes. It can also become a public-facing business problem.

In 2025, Wolf River Electric sued Google over an AI Overview that allegedly told users the company was being sued by the Minnesota Attorney General for deceptive sales practices. According to reporting on the complaint, Wolf River alleged that the cited sources did not actually say what the AI-generated overview claimed they said. Reporting on the lawsuit also described specific alleged business losses tied to the false output, including terminated or threatened contracts after customers saw the AI summary. ²² ²³

Later that year, Robby Starbuck sued Google over chatbot outputs that allegedly included false accusations of sexual assault, fabricated criminal history, and invented court records. Reuters reported that Google later argued Starbuck had intentionally prompted the outputs and had not shown that anyone believed or saw the allegedly defamatory content, while also contesting the required showing of actual malice. ²⁴ ²⁵ ²⁶

Those cases involve allegations and legal arguments, not final rulings on every issue. But they show the same pattern: if a system produces authoritative-sounding falsehoods about real people or businesses, the damage can arrive before the legal system resolves anything.

The Real Pattern

Across all of these examples, the same sequence keeps repeating.

Someone uses AI output as if it were reliable on its own.
The output contains something fabricated, distorted, or unsupported.
The error is not caught before it affects a filing, a record, a customer, a publication, or a reputation.
Then the consequences land. ¹ ³ ⁸ ⁹ ²³ ²⁴

The tool changes. The industry changes. The mechanics vary. The pattern does not.

AI is useful precisely because it can generate fluent, plausible output quickly. That is also why it is risky. The same system that can produce a helpful draft can also produce a confident fabrication in the same tone, with the same formatting, and with just enough surface plausibility to get waved through. ¹⁷ ¹⁸

What This Means for People Building or Buying Software

If you are building products with AI, you need real verification layers. Not just another model evaluating the first one, but checks against source data, known records, validated citations, controlled retrieval, structured guardrails, and human review where the stakes justify it.

If you are using AI in your own work, treat the output like a draft, not a fact base. That applies to code, research, legal writing, medical documentation, content generation, and customer communication.

I use AI every day because it is genuinely useful. But speed only helps when the output is correct. If you skip verification, you are not eliminating work. You are moving it downstream into a more expensive failure.

Most people in these cases were not trying to commit fraud. They were trying to move faster. That is exactly why this matters.

The trap is not using AI.

The trap is trusting it before you check it.

Sources

Mata v. Avianca, Inc. sanctions order (PDF): https://hvbba.org/wp-content/uploads/2025/09/8-PANEL-THREE-Mata-v-Avianca-Inc.pdf ↑
Jones Walker LLP, “From Enhancement to Dependency: What the Epidemic of AI Failures in Law Means for Professionals” (Aug. 19, 2025): https://www.joneswalker.com/en/insights/blogs/ai-law-blog/from-enhancement-to-dependency-what-the-epidemic-of-ai-failures-in-law-means-for.html?id=102l04x ↑
The Colorado Sun, “MyPillow CEO’s lawyers fined for AI-generated court filing in Denver defamation case” (July 7, 2025): https://coloradosun.com/2025/07/07/mike-lindell-attorneys-fined-artificial-intelligence/ ↑
Noland v. Land of the Free, L.P. (Cal. Ct. App. 2025): https://law.justia.com/cases/california/court-of-appeal/2025/b331918.html ↑
Whisper model documentation: https://huggingface.co/openai/whisper-large ↑
Cornell Chronicle, “AI speech-to-text can hallucinate violent language” (June 11, 2024): https://news.cornell.edu/stories/2024/06/ai-speech-text-can-hallucinate-violent-language ↑
Koenecke et al., “Careless Whisper: Speech-to-Text Hallucination Harms” (ACM FAccT 2024): https://dl.acm.org/doi/10.1145/3630106.3658996 ↑
AP, “Researchers say AI transcription tool used in hospitals invents things no one ever said” (Oct. 26, 2024): https://apnews.com/article/ai-artificial-intelligence-health-business-90020cdf5fa16c79ca2e5b6c4c9bbb14 ↑
American Bar Association, “BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot” (Feb. 29, 2024): https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/ ↑
The Guardian, “Air Canada ordered to pay customer who was misled by airline’s chatbot” (Feb. 16, 2024): https://www.theguardian.com/world/2024/feb/16/air-canada-chatbot-lawsuit ↑
Linardon et al., “Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models” (JMIR Mental Health, 2025): https://mental.jmir.org/2025/1/e80371 ↑
GPTZero, “GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers” (Jan. 21, 2026): https://gptzero.me/news/neurips/ ↑
Ansari, “Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025” (arXiv, 2026): https://arxiv.org/abs/2602.05930 ↑
LawNext, “A New Wrinkle in AI Hallucination Cases: Lawyers Dinged for Failing to Detect Opponent’s Fake Citations” (Sept. 16, 2025): https://www.lawnext.com/2025/09/a-new-wrinkle-in-ai-hallucination-cases-lawyers-dinged-for-failing-to-detect-opponents-fake-citations.html ↑
Mavy v. Commissioner of Social Security Administration order: https://docs.justia.com/cases/federal/district-courts/arizona/azdce/2%3A2025cv00689/1428727/18 ↑
ABA Journal, “Confronted with AI hallucinations in filings, one court shows ‘justifiable kindness,’ while another gets tough” (Aug. 19, 2025): https://www.abajournal.com/web/article/court-rejects-monetary-sanctions-for-ai-generated-fake-cases-citing-lawyers-tragic-personal-circumstances ↑
Dahl et al., “Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models” (paper PDF): https://nacmnet.org/wp-content/uploads/Dahl-et-al-Large-Legal-Fictions-Hallucinations-in-LLMs-2401.013012024JAN02-37pp.pdf ↑
Stanford Law School publication page for “Large Legal Fictions”: https://law.stanford.edu/publications/large-legal-fictions-profiling-legal-hallucinations-in-large-language-models/ ↑
PBS NewsHour, “What to know about an AI transcription tool that ‘hallucinates’ medical interactions” (Jan. 25, 2025): https://www.pbs.org/newshour/show/what-to-know-about-an-ai-transcription-tool-that-hallucinates-medical-interactions ↑
Reuters, “It’s too easy to make AI chatbots lie about health information, study finds” (July 1, 2025): https://www.reuters.com/business/healthcare-pharmaceuticals/its-too-easy-make-ai-chatbots-lie-about-health-information-study-finds-2025-07-01/ ↑
EurekAlert summary of the JMIR study (Nov. 17, 2025): https://www.eurekalert.org/news-releases/1106130 ↑
Government Technology / Star Tribune syndication, “Minnesota Solar Company Sues Google Over AI Summary” (June 13, 2025): https://www.govtech.com/public-safety/minnesota-solar-company-sues-google-over-ai-summary ↑
Volokh Conspiracy / Reason, “Large Libel Models: Small Business Sues Google, Claiming AI Overview in Searches Hallucinated Attorney General Lawsuit” (June 11, 2025): https://reason.com/volokh/2025/06/11/large-libel-models-small-business-sues-google-claiming-ai-overview-in-searches-hallucinated-attorney-general-lawsuit/ ↑
The Regulatory Review, “What Starbuck v. Google Reveals About AI Liability” (Dec. 22, 2025): https://www.theregreview.org/2025/12/22/andrews-what-starbuck-v-google-reveals-about-ai-liability/ ↑
Reuters, “Conservative activist sues Google over AI-generated statements” (Oct. 22, 2025): https://www.reuters.com/legal/litigation/conservative-activist-sues-google-over-ai-generated-statements-2025-10-22/ ↑
Reuters, “Google asks court to dismiss conservative influencer’s AI defamation lawsuit” (Nov. 17, 2025): https://www.reuters.com/legal/government/google-asks-court-dismiss-conservative-influencers-ai-defamation-lawsuit-2025-11-17/ ↑