A team from the OmniAI Group at Zhejiang University has released EmbodiedMemory-Bench (EMem-Bench), a benchmark that tests whether embodied AI agents can build memory from long interaction histories and then act on it. The paper was posted to arXiv about three weeks ago and drew Chinese media coverage this weekend. The dataset on Hugging Face showed more than 32,000 downloads as of October 18.
The problem is familiar from household robots: an agent sees a book on the coffee table, finishes other chores, then searches every room when asked to fetch it, or keeps pulling a drawer it already found locked. The team manually reviewed 100 failed trajectories from Gemini-3-Flash and Qwen3-VL-32B on the EB-ALF and EB-Hab suites of EmbodiedBench. Missing fine-grained visual cues accounted for 35% of the failures, failing to record states revealed by interaction for 31%, overlooking changes in the environment for 18%, and failing to reuse past experience for 7%.
EMem-Bench turns these gaps into four task families: Passive Observation, Dynamic Tracking, Interaction Failure and Experience Generalization. Each of its 2,554 executable episodes provides a time-ordered multimodal history followed by a new task. The agent must retrieve the relevant evidence and then navigate, open containers, pick up and place objects in simulation until the goal state is reached. Single-room AI2-THOR scenes are combined into multi-room homes, and distracting but executable interactions are inserted between the key clue and the task. The benchmark spans 1,118 scenes and passed automated checks and human review.
In the full-context setting, Gemini-3-Flash led with a 64.2% average success rate, followed by Gemini-2.5-Pro at 60.8%, Qwen3.6-27B at 58.2% and GPT-5.4 at 56.8%. The paper notes that all but one of the open-source models scored below 45%.
The team also built Embodied-Memorizer (EMem), an external memory system that splits experience into scene, spatial and event memories. With GPT-5.4-mini as the backbone, EMem raised average success from 35.3% to 58.9%, 21.1 points above the next-best memory system, TeleMem. A fine-tuned 8B policy, EMem-8B, reached 46.0%, 20.6 points above its Qwen3-VL-8B base. Simply feeding in more history images did not help: success fell from 36.6% to 26.2% while input tokens per episode rose from 41,770 to 206,489.
The authors note that all results come from simulation and still need to be tested on real robots facing sensor noise and execution errors.
SEE ALSO: UBTECH Details Thinker-WM, a World Model That Unifies Video, Depth and Robot Actions · Tencent Hunyuan Open-Sources Hy Memory 2.0, Adding Working and Code Memory for AI Agents