Australia’s online safety watchdog has revealed sickening new statistics about the use of AI-powered deepfake apps which have the ability to generate realistic sexual imagery of children.
eSafety Commissioner Julie Inman Grant revealed in a parliamentary inquiry on Tuesday that “nudify” apps are being downloaded more than 200,000 times a month by Australians.
Appearing before the Joint Select Committee on Artificial Intelligence in Sydney, Ms Inman Grant issued a stark warning about the ease in which artificial intelligence can be used to produce child sexual abuse material to target Australian schoolchildren.
Ms Inman Grant also confirmed her office has taken legal enforcement action against eight different AI-powered “nudify” apps and services.
The commissioner warned the rapidly developing area of technology is creating increasingly more sophisticated opportunities for criminals to exploit vulnerable groups, such as children.
“These services represent about 200,000 downloads by Australians per month,” Ms Inman Grant told the inquiry.
“We are already seeing AI weaponised to create child sexual exploitation and abuse material with increasing reports of deepfake image-based sexual abuse across Australian schools.”
Further research by the eSafety commission has also found one in five Australian children to have experienced potentially inappropriate or harmful interactions with AI chatbots or companion services.
Concerningly, one in three children had shared personal or potentially sensitive information with the same technology.
Ms Inman Grant said organised crime groups are now using voice cloning and face-cloning technology in expanding sextortion schemes targeting young Australian boys and men.
The commissioner pointed to a recent investigation in Canada where an AI model allegedly created by paedophiles raised concerns of the technology’s ability to facilitate coercive control and abuse over children who develop a dependency on chat companions.
The Canadian investigation found the model to be generating graphic synthetic child sexual abuse material using childhood images of a survivor of this dependency.
“These are real harms affecting real people right now. They’re not just existential threats,” Ms Inman Grant said.
The watchdog warned that this technology left unchecked will result in the same government failures to protect young people during the rise of social media, when regulation struggled to keep up with the advancement of emerging online harms..
“For years, regulation played catch-up with social media harms, after the damage was visible in children’s mental health, in election integrity, and trust in our information ecosystem,” she said.
“And AI is moving faster, and its harms could be much more catastrophic.”
During questioning by the committee, Ms Inman Grant warned Australia could be entering an “age of unreality”, where increasingly realistic AI-generated imagery and video could make it difficult to distinguish fact from fiction.
“We would be living, I think, in a very confusing and polluted information environment where we don’t know what is real and what is not real,” she said.
She also raised the issue of legislative gaps covering non-sexual deepfakes, using the example of two teachers who were falsely depicted kissing despite never having done so.
The commissioner claimed this manipulated video does not meet the threshold for serious adult cyber abuse under the eSafety’s existing framework, despite its potential to cause harm.
Ms Inman Grant said similar regulatory gaps exist in protecting election integrity as AI tools become increasingly capable of producing realistic footage depicting high-profile figures saying or doing things they have not.
The watchdog warns stronger regulation is necessary to ensure technology companies prioritise user safety before releasing new, more advanced, products.
“AI companies should not be left to mark their own homework when their risks are this significant,” Ms Inman Grant said.
“Public trust requires accountability, transparency, and independent oversight.”
Ms Inman Grant rejected suggestions from the committee that stronger regulations would undermine technological innovation and advancements, arguing consumer confidence and assured safety could instead encourage more widespread adoption of the technology.
“Strong safeguards build trust. Trust drives adoption, and adoption drives innovation and productivity,” she said.
The watchdog is already undertaking compliance and enforcement activity against certain AI services under Australia’s existing online safety framework.
However, Ms Inman Grant said regulators needed stronger information-gathering powers to examine algorithms, access platform data and independently test claims made by technology companies.
The eSafety Commission has proposed reforms to digital duty of care practices, allowing regulators to test whether AI companies have adequate safeguards by issuing prompts designed to expose weaknesses in their systems.
Ms Inman Grant said these kinds of tests would be conducted for the specific purpose of determining whether a service had appropriate protections in place to prevent the generation of illegal content.
Currently, the eSafety Commission does not have the power to carry out these tests itself.
Ms Inman Grant said regulators were struggling to match the resources and technological capabilities of major AI companies, warning governments would need greater technical expertise and international co-operation to keep pace..
“This is more than an arms race, they’re way out ahead,” she said.
The commission has backed a coordinated approach across levels of government to avoid overlapping or unnecessarily burdensome regulatory responsibilities.
Ms Inman Grant said proposed digital duty of care reforms would help shift responsibility towards technology companies identifying and reducing foreseeable risks before harm occurred.
“Trust, however, cannot be an afterthought. It must be earned and designed into the products and services Australians use every day,” she said.