“I COMPLETELY FABRICATED THOSE DETAILS”  

When ChatGPT Gets the Law Wrong

Artificial intelligence is changing the way we work, research and obtain information. But a recent Australian court decision provides a striking warning about what can happen when AI-generated legal information is relied upon without being properly checked.

In Asif v Minister for Immigration and Citizenship [2026] FedCFamC2G 1402, a self-represented litigant used ChatGPT to assist in preparing submissions for his court case.

The problem? He could not properly explain or substantiate the legal arguments ChatGPT had generated.

Judge Humphreys described the submissions as unnecessarily lengthy and repetitive, with little substantive legal argument. There were also troubling inaccuracies, including a reference to a paragraph that did not exist and cases that did not support the propositions attributed to them.

But perhaps the most extraordinary part of the judgment came from the Judge’s own experiment with ChatGPT.

His Honour had previously asked ChatGPT about one of his own judgments. ChatGPT confidently provided information about the case which was simply untrue. When challenged, ChatGPT responded:

“I completely fabricated those details.”

The Court’s message was clear: AI can “hallucinate”, producing information that sounds authoritative and convincing but is, in fact, incorrect.

And there can be real consequences.

The applicant’s case was dismissed and he was ordered to pay the Minister’s costs, fixed at $9,097.93. The Court also warned that “sterner action may be needed” in the future, including potentially refusing to accept submissions filed contrary to court orders and apparently generated by AI in response to another party’s case.

So, should we stop using AI?

Not necessarily.

AI can be an extremely useful tool. The danger lies in assuming that because an AI-generated answer sounds confident, detailed and convincing, it must be correct.

This is particularly important when dealing with legal issues. A court case may turn upon the precise wording of legislation, the actual reasoning in a judgment, procedural requirements and the individual facts of the case.

For lawyers, there is an additional safeguard: we owe professional duties to the Court, including duties of candour and accuracy. Judge Humphreys noted that lawyers who rely upon false or hallucinated AI-generated material may face disciplinary consequences.

The takeaway? Use AI as a tool, not as your lawyer.

When the outcome matters, there remains no substitute for proper legal advice, careful research and human professional judgment.

Asif v Minister for Immigration and Citizenship [2026] FedCFamC2G 1402 (22 July 2026).

To discuss your matter with one of our experienced team, contact us today on 8525 2700, or click here to request an appointment.

Article by Riccarda Stock

Image created in Canva