Germany’s Neura Robotics has unveiled a foundation model to help robots make decisions using touch, sound and other sensory inputs alongside vision, in a bid to improve the machines’ perception capabilities.
The new model, Neuron AI, combines 3D vision, tactile and force data, audio, robot state and language in a perception-language-action model. Neura said the design also incorporates “memory,” enabling robots to retain information about objects and previous actions even when those objects are no longer visible.
Neura is planning a live demonstration of Neuron AI at Nvidia’s GTC conference in Berlin later this week. At the demo, Neura’s humanoid robot 4NE1 will perform an eleven-minute laundry-handling sequence, trained using the new model.
The approach differs from traditional physical AI systems that primarily rely on vision-language-action (VLA) models, which use visual information to learn to perform tasks.
Related:Robots can't talk to each other yet -- that's a big AI bottleneck
According to Neura, Neuron AI uses three sources of data to train robots: real-world robot interactions, simulated tasks and environments, and human demonstrations captured with sensor-equipped suits.
The company collects the sensory and movement data at its Neura Gyms, which provide infrastructure for robot training models.
Yet while combining multiple sensory inputs could help robots handle tasks that cameras alone cannot, the approach poses a training challenge: collecting sufficient suitable data to teach models to understand and respond to those inputs.
“Most physical AI stacks are focused on VLA models, which learn from ‘seeing,’” said Rueben Scriven, senior research manager at industrial automation and robotics research market firm Interact Analysis. “What Neura is betting on is that models that can be trained on different sensory inputs will perform better.”
The approach could also complement Neura’s vertically integrated sensor stack, potentially giving the company an advantage over competitors.
However, Scriven questioned whether Neura could collect sufficient data to effectively train the model across so many sensory inputs.
“There’s already a limit on well-annotated video data, let alone multi-modal input data,” he said. “Therefore, the real question is whether data collection across its robotic gyms will be enough to train the models.”
Training robots beyond vision
Neura argues that additional sensory inputs are important for tasks in which visual information alone cannot reliably determine what is happening. When a robot opens a lock, only sound and touch confirm that it is really open, according to the company.
Related:China leads dexterous hand supply for humanoids
The model’s memory function is intended to help robots maintain context as objects move out of view, and as the robot’s actions change its surroundings.
The company hasn’t yet disclosed how much the extra sensory inputs affect performance, or how much training data is needed to achieve reliable results.
About the Author
Contributing Writer
Scarlett Evans is a freelance writer with a focus on emerging technologies and the minerals industry. Previously, she served as assistant editor at IoT World Today, where she specialized in robotics and smart city technologies. Scarlett also has a background in the mining and resources sector, with experience at Mine Australia, Mine Technology and Power Technology. She joined Informa in April 2022 before transitioning to freelance work.