A new hybrid optical-neural system can detect artificially generated “deepfake” videos with up to 97.8% accuracy. The technique, which was developed by a team at the University of California, Los Angeles (UCLA) in the US, can simultaneously process 15 or more independent video streams and is less energy-hungry than existing methods with similar levels of performance.
As artificial intelligence (AI) has improved, so has the quality of AI-generated videos. While many early models churned out content with glaring errors such as extra fingers and nonsense text, more up-to-date versions of generative diffusion models (for example) are now capable of producing highly realistic images and videos. Indeed, a recent study found that humans correctly identify deepfakes only around 50% of the time. “While AI technologies open new possibilities for content creation, they also raise growing concerns about misinformation, identity manipulation and the erosion of trust in digital media,” says Aydogan Ozcan, who led the new study. “Efficient automated detection systems are therefore sorely needed.”
The problem, Ozcan explains, is that existing systems rely on computationally intensive neural networks that are slow and use lots of energy. That makes it hard to scale them up to the levels required for large-scale video processing tasks. Detection systems based on deep learning are also inherently vulnerable to so-called adversarial attacks in which slight alterations to fake videos can fool the detector into labelling them as authentic.
Let us use light
To overcome these problems, Ozcan and colleagues decided to explore whether some of this computation could instead be performed using the physical properties of light. In their work, which is described in eLight, they designed a lightweight digital encoder that extracts spatial and temporal optical information from each video and converts it into a phase pattern displayed on a programmable spatial light modulator. The encoded optical wavefront then propagates through a free-space-based optical decoder. At the system output, paired optical detectors generate an “authenticity score” for each video.
The researchers tested their system using face-swapped videos, real-world deepfake recordings and AI-generated videos. In one test based on the Celeb-DF database, which contains identity-based face-swap manipulations alongside the corresponding real videos, they showed that their system could simultaneously process batches of 15 videos in a single step with an average detection accuracy of 97.79%. For this test, the system had a sensitivity of 99.86% (meaning that 0.14% of fake videos were falsely labelled as genuine), and a specificity of 95.72% (meaning that 4.18% of real videos were incorrectly judged to be fakes).
In a similar test of videos generated using Google’s VEO-3 model, the optical processor reached 94.80% accuracy and 97.61% sensitivity in the team’s 15 parallel video experiments. According to Ozcan, this result shows that after lightweight fine-tuning, the system can adapt to newer-style generative-AI pipelines that lack many of the artifacts found in earlier deepfakes.
“Importantly,” he tells Physics World, “this efficiency does not come at the expense of robustness: the optical decoder is also resilient to noise, image compression and physical misalignment as well as adversarial attacks. What makes this particularly exciting is that optical computation simultaneously provides parallelism, energy efficiency and an additional physical barrier against adversarial manipulation – properties difficult to achieve together with conventional digital hardware.”
A high-throughput, first-stage screening system
Ozcan and his colleagues envision the technology primarily as a high-throughput, first-stage screening system for large-scale media platforms or data centres. Rather than replacing sophisticated digital forensic models, the optical processor could rapidly examine enormous numbers of videos with high sensitivity and comparatively low energy consumption, forwarding suspicious content to more computationally intensive digital models for detailed verification, they say.
Beyond deepfake detection, Ozcan says the work shows that embedding AI computation within optical hardware can create new combinations of efficiency and security, since important model parameters are physically embodied in the optical system and are therefore considerably harder to extract or reproduce. Similar principles could ultimately benefit other high-throughput, security-critical AI inference tasks, he adds.
The eyes have it: how to spot the difference between a deepfake portrait and a real picture
Read more
The team now plans to scale up and adapt the system’s architecture. For example, optical modulators with a larger aperture should allow for substantially more videos to be multiplexed. Second, faster modulators, such as MEMS-based phase-light modulators, could further reduce latency and energy consumption per inference.
Finally, Ozcan notes that it will be important to adapt the technology rapidly so that it can handle new generations of synthetic video. The system’s performance on VEO-3 generated videos is, he says, a good early indication that it is possible to develop optical-neural systems that can be efficiently updated (rather than completely redesigned) as generative AI evolves.