Spot AI astrology before it becomes a court sanction

Non-advice notice: This article is legal-information and verification-workflow material, not legal advice. It is designed to help readers identify filing-risk signals and trace authorities before relying on AI-generated legal text.

• Legal-background review signal: litigation-risk editorial review for court-filing, citation-verification, and Rule 11 issues.
• Last verified: 26 August 2026 (UTC).
• Case-count note: AI hallucination and sanction counts are live snapshots. Different trackers report different totals at different dates; treat every count below as date-stamped, not permanent.

The dangerous legal-AI output is not the one that looks ridiculous. The dangerous output is the polished paragraph that reads as though a careful associate wrote it, cites cases in the right cadence, and lands in a draft brief at the point when everyone is tired. That is the useful sense of “AI astrology”: confident structure, convincing tone, and no verifiable anchor.

The National Center for State Courts gives the failure mode its professional seriousness: generative AI can produce text that “sounds right rather than is right.” For court filings, that difference is not stylistic. It is the difference between a proposition that can be opened, read, and checked by a judge or opponent, and a proposition that merely feels familiar until someone tries to Shepardize it at 11 p.m. [1]

The live record is already large enough to change the workflow

The best starting point is not a model architecture diagram. It is the sanction and error record. Damien Charlotin’s AI Hallucination Cases Database reported 1,934 decisions as of its 19 August 2026 update, including 1,325 U.S. decisions, 768 matters involving lawyers, 1,115 involving pro se litigants, 1,739 fabricated-case-law findings, and 522 false-quote findings. Those figures are not an eternal total; they are a dated snapshot of a live database. They are still enough to make one point hard to avoid: courts are no longer dealing with AI hallucinations as a hypothetical risk. [2]

The disagreement between snapshots is not a nuisance to be edited away. It is part of the record. Charlotin, HAQQ, and other secondary counts are not measuring identical universes at identical moments. A filing-risk workflow that treats a secondary tracker as if it were a court order is repeating the same unanchored habit it is supposed to prevent.

HAQQ’s tracker is still worth watching because it shows acceleration and consequence. It reported growth from about 200 cases in mid-2025 to 1,598 by 9 June 2026, described the pace as roughly eight new cases per day, and identified Q1 2026 U.S. penalties above $145,000. It also pointed to a reported single-matter penalty around $109,700 in Couvrette v. Wisnovsky and to Withers v. City of Aberdeen, where the account describes a canceled trial, two-year district suspensions, and fines for every lawyer of record. Those are serious signals, but the penalty figures should be re-verified against the primary order or primary reporting before they are quoted in a memo, CLE deck, or client alert. [3]

Good faith is not the verification standard

The legal problem with AI astrology is that it invites a human to mistake subjective confidence for objective inquiry. Rule 11 does not ask whether the lawyer felt misled by a plausible tool. It asks whether the pre-filing conduct was reasonable. Thomson Reuters Institute’s discussion of recent hallucination matters puts Mata v. Avianca at the origin point of many lawyers’ awareness: in June 2023, the court imposed a $5,000 sanction after fabricated cases appeared in a filing. [4]

The more durable warning is older than generative AI. Thomson Reuters quotes Deghani v. Castro for the principle that an attorney cannot avoid Rule 11 consequences by having “an empty head and a pure heart.” That sentence should be taped to every AI-use policy that touches court filings. Belief, pressure, delegation, and lack of bad faith may matter to tone and discipline. They do not substitute for opening the authority and checking whether it says what the filing says it says. [4]

Supervisor exposure belongs in the same conversation. The Thomson Reuters Institute article also discusses ABA Formal Opinion 512 and supervisory-responsibility examples including Tan Hai Peng v. Tan Cheong Joo and Ayinde v. Haringey. The practical implication is uncomfortable but clear enough: the cleanup burden does not stay with the person who pasted the AI text. It travels to the lawyer who signs, supervises, submits, or fails to stop the filing. [4]

That is why “the tool sounded authoritative” is not a useful defense. It is a description of the trap.

Research tools can still produce unanchored answers

The benchmark evidence matters because it removes the comforting story that hallucination is only a free-chatbot problem. Stanford HAI reported that general-purpose chatbots hallucinated on legal queries at rates between 58% and 82%. More relevant for law offices, it also reported that legal research products still returned incorrect information: more than 17% for Lexis+ AI and Ask Practical Law AI, and more than 34% for Westlaw AI-Assisted Research in the benchmark discussed. Those figures should not be turned into a simple permanent vendor ranking; they are benchmark results tied to particular systems, tasks, and dates. They are nevertheless a warning against treating research-product branding as verification. [5]

The Yale ISPS/JELS study reached the same operationally important point from another angle: paid retrieval-augmented legal AI tools still hallucinated at rates between 17% and 33%. “Retrieval-augmented” is not the same as “court-ready.” It means the system is designed to retrieve and use source material. It does not mean every proposition in the answer has been matched to a source that supports it. [6]

Dahl et al.’s Large Legal Fictions study is broader and more severe. It tested GPT-4, GPT-3.5, PaLM 2, and Llama 2 across more than 800,000 verifiable legal questions and reported hallucination rates from 58% to 88%. That study is most useful here not as a prediction about every current legal product, but as evidence that legal-sounding output can fail at scale when the task requires verifiable legal truth rather than fluent legal prose. [7]

The misgrounded answer is the closest thing to AI astrology

Stanford’s most useful distinction is not simply right versus wrong. It separates incorrect answers from misgrounded ones. A misgrounded answer may sound responsive, may cite something that exists, and may even gesture toward the right doctrinal neighborhood, while the cited source does not actually support the proposition. Stanford described that category as potentially “more pernicious.” For litigation risk, it is easy to see why: the answer passes the first human smell test and fails only when someone performs the slower act of matching sentence to source. [5]

That is the astrology resemblance in its narrow, useful form. A horoscope can feel specific because it has structure, tone, and familiar categories. A legal-AI paragraph can feel researched because it has issue statements, case names, parentheticals, and measured caveats. In both cases, the question is whether the apparent precision is anchored to something independently inspectable.

The confidence problem makes this harder. The AI Law Librarians roundup, discussing the emerging science, describes findings including jurisdictional degradation in “Place Matters”: reported error rates of 45% for Los Angeles, 55% for London, 61% for Sydney, and 100% for one local statute in the covered study. Because that discussion is second-hand through the roundup, those figures should be traced to the underlying study before being used as primary support. Even as a secondary signal, the lesson is familiar to anyone who has checked local rules: the narrower the jurisdictional and procedural question, the less useful a confident general answer becomes. [8]

A brief technical analogy from Artificial Inquiry helps explain why a vibe check fails. The piece contrasts fluent chatbot output with astrological calculations that require exact positions, noting that the Moon moves about 13 degrees per day and the Ascendant about one degree every four minutes. The point for legal work is not astrology; it is that some tasks require calculation or source-matching, not plausible continuation of a pattern. A filing is one of those tasks. [9]

The pre-filing astrology test

This site’s “astrology test” is a verification workflow synthesized from the NCSC warning, the Stanford and Yale benchmark evidence, Rule 11 sanction records, ABA Formal Opinion 512, and the live Risk Digest record. It is not a standard announced by any one court or vendor. It is a practical way to move a draft from “sounds right” to “can be inspected.”

1. Open every cited authority. Do not rely on the AI interface, a generated citation string, or a secondary tracker. The case, statute, rule, regulation, order, or article must be accessible in a source the court, opponent, or client can independently inspect.
2. Find every quotation in the source. The quotation must appear in the cited material, with the same words and a defensible pincite. If the language is paraphrased, mark it as a paraphrase and confirm that the source actually supports it.
3. Match every legal proposition to authority. A real case is not enough. The cited authority must support the sentence for which it is cited, not merely discuss the same topic.
4. Confirm jurisdiction, court level, date, and procedural posture. A proposition from a different jurisdiction, an unpublished order, a dissent, a vacated decision, a pleading, or a distinguishable procedural setting may be useless or harmful in the filing where the AI placed it.
5. Update and negative-check the authority. Run the normal citator, statutory update, rule update, and local-rule check. AI verification does not replace Shepardizing, KeyCiting, or jurisdiction-specific updating.
6. Document the human check before filing. The record should show who opened the source, who checked the quotation, who confirmed the proposition, and when the check occurred. If a supervising lawyer signs the filing, the verification record should be available before signature, not reconstructed after an order to show cause.

The workflow is intentionally slower than copying a polished answer. That is the point. It puts the burden back where litigation rules already place it: on the human filer and the supervisory chain, not on the tone of the generated text.

Where the check should lead next

If a suspicious filing, tool answer, or policy memo triggered this review, the next step is record-level checking. Use Risk Digest entries for matters such as Mata v. Avianca, Withers v. City of Aberdeen, Couvrette v. Wisnovsky, and Kaur v. Desso to move from secondary descriptions to the orders and docket materials behind them.

For policy work, route the same issue through Regulation & Ethics for Rule 11 and ABA guidance, Tool Reliability Evaluations for benchmark evidence, and Verification Workflows for reusable pre-filing checks. AI output is not unsafe because it sounds strange. It is unsafe when it sounds complete while remaining unanchored.

References

1. A legal practitioner’s guide to AI & hallucinations, National Center for State Courts
2. AI Hallucination Cases Database, Damien Charlotin, last updated 19 Aug 2026
3. AI Hallucination Cases: The 1,598-Case Sanctions Tracker, HAQQ, 9 Jun 2026
4. GenAI hallucinations are still pervasive in legal filings, but better lawyering is the cure, Thomson Reuters Institute
5. AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries, Stanford HAI
6. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Yale ISPS/JELS
7. Large Legal Fictions, Journal of Legal Analysis
8. What the Science Says About Hallucinations in Legal Research, AI Law Librarians, 19 Feb 2026
9. The Stars Don’t Lie (But Your Chatbot Does), Artificial Inquiry