The authors of a report testing the technology underpinning Australia’s social media ban have conceded ChatGPT was used in editing, but denied a number of citation errors in the report were due to AI hallucinations.
The $۳.۴۸m age assurance technology trial, run by the UK-based Age Check Certification Scheme (ACCS) last year, tested various types of technology that could be used by social media platforms as part of Australia’s under-۱۶ social media ban.
The communications minister, Anika Wells, heralded the report as showing “many effective options” for checking people’s ages, and it paved the way for the ban to come into effect in December last year.
Most of the report is based on tests the authors did of the various age-verification technologies currently available, but one chapter covers “emerging technologies” and cites a number of journal papers about potential new ways to check ages.
An ongoing Senate inquiry seeking to further strengthen the social media ban law received a submission highlighting at least two citation errors in the report. The “citations … appear to be AI hallucinated, rather than being based on real sources,” it said.
When asked about the allegation by Guardian Australia this month, a spokesperson for ACCS initially denied the use of AI in the report, but later conceded that AI had been used to rewrite paragraphs more succinctly.
“We did not use AI in the generation of the report or the cited materials … Each one of them were checked as genuine links and reports and that they were relevant to the specific issue being cited,” they said.
However, Guardian Australia’s own analysis of the report identified six references in the same section that contained errors.
The errors include: Digital Object Identifiers (DOIs – unique combinations of letters and numbers which are associated with specific academic articles) that link to papers that do not exist; DOIs that point to incorrect papers; author name, journal and publication year combinations that do not match any known references; and DOI links to papers that did not say what the report suggested they said.
The spokesperson later provided clarifications and new references to the research ACCS had relied on, but this response too contained errors.
A list of journal articles intended to correct those in the report included years, authors and journal titles not matching the original citations in the report.
One article ACCS cited was apparently accessed in March ۲۰۲۵. However, one of the lead authors of this study confirmed to the Guardian that their article was not publicly accessible until it was published in June ۲۰۲۵. They also confirmed that the person originally cited by ACCS as the paper’s lead author, “Jamil”, had never been on the paper, and that the ACCS report had summarised their paper incorrectly.
The spokesperson provided further clarification regarding another reference that could not be found in the journal cited – “Weber et al. ۲۰۱۱” – by saying it was cited in a study by Monash University, which they provided to the Guardian. However there was no reference attributed to Weber in the Monash paper ACCS sent.
The spokesperson only conceded AI had been used after the Guardian identified four links in part E and part K of the report that contained metadata identifying that the source of the link was ChatGPT.
They maintained that by including the metadata of ChatGPT in the links the organisation had disclosed its use, and given the links were correct, they did not see that as a problem.
They said AI had been used to rewrite some paragraphs more succinctly, but was not used for originating research or for the generation of the report.
“All of the links were checked and verified and, whilst I apologise if there is an inadvertent error, this was all done by human verification.”
after newsletter promotion
The federal government has previously taken a dim view of contractors using AI in reports to government. Deloitte last year refunded part of its $۴۴۰,۰۰۰ contract with the government after errors were identified, and the consultant firm admitted to using AI.
Questions to Wells’ office were redirected to her department.
“The department is examining the concerns raised below and will engage with ACCS as appropriate,” a spokesperson for the department said last Friday.
In a hearing for the Senate inquiry on Friday, department officials said they met with ACCS, who claimed that the errors were due to links breaking that had previously worked. The officials have not independently verified this, stating it would be “very difficult” to go back and check.
They said it was a “handful of errors” in ۲۶ pages of citations for a ۱,۰۰۰-page report.
“[ACCS] assured us that the links were all checked at the time that the document was published, and they all worked then,” the department’s first assistant secretary, Sarah Vandenbroek, said. “So, something may have changed in the interim. [They] did also assure us that while some of the links may have been broken, though the source documents do still exist, they are still valid.”
Prof Christian Downie, from the school of regulation and global governance at the Australian National University, who has previously examined submissions to government that contained AI hallucinations, said if reports or submissions to government are found to have false citations in them – AI or otherwise – it could lead to bad decisions, and “erode public confidence and trust” in institutions.
“One would assume the government is already thinking about this, and that is going to have to require contractual arrangements and presumably penalties to discourage this type of behaviour,” he said.
Fatima Payman, an independent senator, said: “The saga of last year’s Deloitte AI slop report should have marked the end of slack referencing in reports by government contractors.”
“The government must demand an apology and refund from the contractor behind the report,” she said.
