Judge Victoria Kolakowski sensed something was wrong with Exhibit 6C.
Submitted by the plaintiffs in a California housing dispute, the video showed a witness whose voice was disjointed and monotone, her face fuzzy and lacking emotion. Every few seconds, the witness would twitch and repeat her expressions.
Kolakowski, who serves on California’s Alameda County Superior Court, soon realized why: The video had been produced using generative artificial intelligence. Though the video claimed to feature a real witness — who had appeared in another, authentic piece of evidence — Exhibit 6C was an AI “deepfake,” Kolakowski said.
The case, Mendones v. Cushman & Wakefield, Inc., appears to be one of the first instances in which a suspected deepfake was submitted as purportedly authentic evidence in court and detected — a sign, judges and legal experts said, of a much larger threat.
Citing the plaintiffs’ use of AI-generated material masquerading as real evidence, Kolakowski dismissed the case on Sept. 9. The plaintiffs sought reconsideration of her decision, arguing the judge suspected but failed to prove that the evidence was AI-generated. Judge Kolakowski denied their request for reconsideration on Nov. 6. The plaintiffs did not respond to a request for comment.
With the rise of powerful AI tools, AI-generated content is increasingly finding its way into courts, and some judges are worried that hyperrealistic fake evidence will soon flood their courtrooms and threaten their fact-finding mission.
NBC News spoke to five judges and 10 legal experts who warned that the rapid advances in generative AI — now capable of producing convincing fake videos, images, documents and audio — could erode the foundation of trust upon which courtrooms stand. Some judges are trying to raise awareness and calling for action around the issue, but the process is just beginning.
“The judiciary in general is aware that big changes are happening and want to understand AI, but I don’t think anybody has figured out the full implications,” Kolakowski told NBC News. “We’re still dealing with a technology in its infancy.”
Prior to the Mendones case, courts have repeatedly dealt with a phenomenon billed as the “Liar’s Dividend,” — when plaintiffs and defendants invoke the possibility of generative AI involvement to cast doubt on actual, authentic evidence. But in the Mendones case, the court found the plaintiffs attempted the opposite: to falsely admit AI-generated video as genuine evidence.
Tune in to Hallie Jackson NOW, Wednesday at 5 PM EST, for more about this story.
Judge Stoney Hiljus, who serves in Minnesota’s 10th Judicial District and is chair of the Minnesota Judicial Branch’s AI Response Committee, said the case brings to the fore a growing concern among judges.
“I think there are a lot of judges in fear that they’re going to make a decision based on something that’s not real, something AI-generated, and it’s going to have real impacts on someone’s life,” he said.
Many judges across the country agree, even those who advocate for the use of AI in court. Judge Scott Schlegel serves on the Fifth Circuit Court of Appeal in Louisiana and is a leading advocate for judicial adoption of AI technology, but he also worries about the risks generative AI poses to the pursuit of truth.
“My wife and I have been together for over 30 years, and she has my voice everywhere,” Schlegel said. “She could easily clone my voice on free or inexpensive software to create a threatening message that sounds like it’s from me and walk into any courthouse around the country with that recording.”
“The judge will sign that restraining order. They will sign every single time,” said Schlegel, referring to the hypothetical recording. “So you lose your cat, dog, guns, house, you lose everything.”
Judge Erica Yew, a member of California’s Santa Clara County Superior Court since 2001, is passionate about AI’s use in the court system and its potential to increase access to justice. Yet she also acknowledged that forged audio could easily lead to a protective order and advocated for more centralized tracking of such incidents. “I am not aware of any repository where courts can report or memorialize their encounters with deep-faked evidence,” Yew told NBC News. “I think AI-generated fake or modified evidence is happening much more frequently than is reported publicly.”
Yew said she is concerned that deepfakes could corrupt other, long-trusted methods of obtaining evidence in court. With AI, “someone could easily generate a false record of title and go to the county clerk’s office,” for example, to establish ownership of a car. But the county clerk likely will not have the expertise or time to check the ownership document for authenticity, Yew said, and will instead just enter the document into the official record.
“Now a litigant can go get a copy of the document and bring it to court, and a judge will likely admit it. So now do I, as a judge, have to question a source of evidence that has traditionally been reliable?” Yew wondered.
Though fraudulent evidence has long been an issue for the courts, Yew said AI could cause an unprecedented expansion of realistic, falsified evidence. “We’re in a whole new frontier,” Yew said.
Santa, Calif., Clara County Superior Court Judge Erica Yew.Courtesy of Erica Yew
Schlegel and Yew are among a small group of judges leading efforts to address the emerging threat of deepfakes in court. They are joined by a consortium of the National Center for State Courts and the Thomson Reuters Institute, which has created resources for judges to address the growing deepfake quandary.
The consortium labels deepfakes as “unacknowledged AI evidence” to distinguish these creations from “acknowledged AI evidence” like AI-generated accident reconstruction videos, which are recognized by all parties as AI-generated.
Earlier this year, the consortium published a cheat sheet to help judges deal with deepfakes. The document advises judges to ask those providing potentially AI-generated evidence to explain its origin, reveal who had access to the evidence, share whether the evidence had been altered in any way and look for corroborating evidence.
In April 2024, a Washington state judge denied a defendant’s efforts to use an AI tool to clarify a video that had been submitted.
Beyond this cadre of advocates, judges around the country are starting to take note of AI’s impact on their work, according to Hiljus, the Minnesota judge.
“Judges are starting to consider, is this evidence authentic? Has it been modified? Is it just plain old fake? We’ve learned over the last several months, especially with OpenAI’s Sora coming out, that it’s not very difficult to make a really realistic video of someone doing something they never did,” Hiljus said. “I hear from judges who are really concerned about it and who think that they might be seeing AI-generated evidence but don’t know quite how to approach the issue.”
Hiljus is currently surveying state judges in Minnesota to better understand how generative AI is showing up in their courtrooms.
To address the rise of deepfakes, several judges and legal experts are advocating for changes to judicial rules and guidelines on how attorneys verify their evidence. By law and in concert with the Supreme Court, the U.S. Congress establishes the rules for how evidence is used in lower courts.
One proposal crafted by Maura R. Grossman, a research professor of computer science at the University of Waterloo and a practicing lawyer, and Paul Grimm, a professor at Duke Law School and former federal district judge, would require parties alleging that the opposition used deepfakes to thoroughly substantiate their arguments. Another proposal would transfer the duty of deepfake identification from impressionable juries to judges.
The proposals were considered by the U.S. Judicial Conference’s Advisory Committee on Evidence Rules when it conferred in May, but they were not approved. Members argued “existing standards of authenticity are up to the task of regulating AI evidence.” The U.S. Judicial Conference is a voting body of 26 federal judges, overseen by the chief justice of the Supreme Court. After a committee recommends a change to judicial rules, the conference votes on the proposal, which is then reviewed by the Supreme Court and voted upon by Congress.
Despite opting not to move the rule change forward for now, the committee was eager to keep a deepfake evidence rule “in the bullpen in case the Committee decides to move forward with an AI amendment in the future,” according to committee notes.
Grimm was pessimistic about this decision given how quickly the AI ecosystem is evolving. By his accounting, it takes a minimum of three years for a new federal rule on evidence to be adopted.
The Trump administration’s AI Action Plan, released in July as the administration’s road map for American AI efforts, highlights the need to “combat synthetic media in the court system” and advocates for exploring deepfake-specific standards similar to the proposed evidence rule changes.
Yet other law practitioners think a cautionary approach is wisest, waiting to see how often deepfakes are really passed off as evidence in court and how judges react before moving to update overarching rules of evidence.
Jonathan Mayer, the former chief science and technology adviser and chief AI officer at the U.S. Justice Department under President Joe Biden and now a professor at Princeton University, told NBC News he routinely encountered the issue of AI in the court system: “A recurring question was whether effectively addressing AI abuses would require new law, including new statutory authorities or court rules.”
“We generally concluded that existing law was sufficient,” he said. However, “the impact of AI could change — and it could change quickly — so we also thought through and prepared for possible scenarios.”
In the meantime, attorneys may become the first line of defense against deepfakes invading U.S. courtrooms.
Louisiana Fifth Circuit Court of Appeal Judge Scott Schlegel.Courtesy of Scott Schlegel
Judge Schlegel pointed to Louisiana’s Act 250, passed earlier this year, as a successful and effective way to change norms about deepfakes at the state level. The act mandates that attorneys exercise “reasonable diligence” to determine if evidence they or their clients submit has been generated by AI.
“The courts can’t do it all by themselves,” Schlegel said. “When your client walks in the door and hands you 10 photographs, you should ask them questions. Where did you get these photographs? Did you take them on your phone or a camera?”
“If it doesn’t smell right, you need to do a deeper dive before you offer that evidence into court. And if you don’t, then you’re violating your duties as an officer of the court,” he said.
Daniel Garrie, co-founder of cybersecurity and digital forensics company Law & Forensics, said that human expertise will have to continue to supplement digital-only efforts.
“No tool is perfect, and frequently additional facts become relevant,” Garrie wrote via email. “For example, it may be impossible for a person to have been at a certain location if GPS data shows them elsewhere at the time a photo was purportedly taken.”
Metadata — or the invisible descriptive data attached to files that describe facts like the file’s origin, date of creation and date of modification — could be a key defense against deepfakes in the near future.
For example, in the Mendones case, the court found the metadata of one of the purportedly-real-but-deepfaked videos showed that the plaintiffs’ video was captured on an iPhone 6, which was impossible given that the plaintiff’s argument required capabilities only available on an iPhone 15 or newer.
Courts could also mandate that video- and audio-recording hardware include robust mathematical signatures attesting to the provenance and authenticity of their outputs, allowing courts to verify that content was recorded by actual cameras.
Such technological solutions may still run into critical stumbling blocks similar to those that plagued prior legal efforts to adapt to new technologies, like DNA testing or even fingerprint analysis. Parties lacking the latest technical AI and deepfake know-how may face a disadvantage in proving evidence’s origin.
Grossman, the University of Waterloo professor, said that for now, judges need to keep their guard up.
“Anybody with a device and internet connection can take 10 or 15 seconds of your voice and have a convincing enough tape to call your bank and withdraw money. Generative AI has democratized fraud.”
“We’re really moving into a new paradigm,” Grossman said. “Instead of trust but verify, we should be saying: Don’t trust and verify.”
