AI-powered apps capable of generating sexualised deepfake images are being downloaded about 200,000 times a month by Australians, the nation’s online safety watchdog has told a parliamentary inquiry.
eSafety Commissioner Julie Inman Grant revealed the figure on Tuesday as she warned artificial intelligence was being weaponised to produce child sexual abuse material and target Australian schoolchildren.
Appearing before the Joint Select Committee on Artificial Intelligence in Sydney, Ms Inman Grant said her office had taken enforcement action against eight AI-powered nudify services.
“These services represent about 200,000 downloads by Australians per month,” she said.
The commissioner warned the rapidly developing technology was creating increasingly sophisticated opportunities for criminals to exploit vulnerable Australians.
“We are already seeing AI weaponised to create child sexual exploitation and abuse material with increasing reports of deepfake image-based sexual abuse across Australian schools,” she said.
New research from eSafety also found one in five Australian children had experienced potentially inappropriate or harmful interactions with AI chatbots or companions.
One in three children had shared personal or potentially sensitive information with the technology.
Ms Inman Grant said organised criminals were using voice cloning and face-swapping technology in sextortion schemes targeting young Australian boys and men.
She also raised concerns about AI-powered companions designed to encourage dependency among children, alongside the technology’s potential to facilitate coercive control and domestic abuse.
The commissioner pointed to a recent investigation in Canada involving an artificial intelligence model allegedly created by pedophiles to generate graphic synthetic child sexual abuse material using childhood images of a survivor.
“These are real harms affecting real people right now. They’re not just existential threats,” she said.
Ms Inman Grant warned governments risked repeating the mistakes made during the rise of social media, when regulation struggled to keep pace with emerging online harms.
“For years, regulation played catch-up with social media harms, after the damage was visible in children’s mental health, in election integrity, and trust in our information ecosystem,” she said.
“And AI is moving faster, and its harms could be much more catastrophic.”
The commissioner argued stronger regulation was necessary to ensure technology companies prioritised user safety before releasing new products.
“AI companies should not be left to mark their own homework when their risks are this significant,” she said.
“Public trust requires accountability, transparency, and independent oversight.”
She rejected suggestions stronger safeguards would necessarily undermine technological innovation, arguing consumer confidence could instead encourage wider adoption.
“Strong safeguards build trust. Trust drives adoption, and adoption drives innovation and productivity,” she said.
The watchdog is already undertaking compliance and enforcement activity against certain AI services under Australia’s existing online safety framework.
However, Ms Inman Grant said regulators needed stronger information-gathering powers to examine algorithms, access platform data and independently test claims made by technology companies.
She also backed a co-ordinated approach across government to avoid overlapping regulatory responsibilities.
The commissioner said proposed digital duty of care reforms would help shift responsibility towards technology companies identifying and reducing foreseeable risks before harm occurred.
“Trust, however, cannot be an afterthought. It must be earned and designed into the products and services Australians use every day,” she said.