RoboBenchMart: Benchmarking Robots in Retail Environment

Most existing robotic manipulation benchmarks focus on tabletop or household scenarios. While these setups have driven impressive progress, it remains unclear whether generalist VLAs that excel there can truly generalize to domains with different geometry, semantics, and workflows. We introduce RoboBenchMart, an open-source simulated benchmark targeting retail dark-store environments, where a mobile manipulator must perform complex manipulation tasks with diverse grocery items. This setting presents significant challenges, including dense object clutter and varied spatial configurations, with items positioned at different heights, depths, and in close proximity. By targeting on the retail domain, our benchmark addresses a setting with strong potential for near-term automation impact. Using generated trajectories, we model a standard, realistic fine-tuning setup for current generalist VLAs and evaluate several state-of-the-art models. We find that they still struggle even on common retail tasks, indicating that these models are not yet truly general across domains. To support further research, we release the RoboBenchMart suite, which includes a procedural store layout generator, a trajectory generation pipeline, evaluation tools, and fine-tuned baseline models.

Real-world evaluation is also difficult to standardize, often requiring human resets and suffering from environment variability. As a result, simulation-based benchmarks have become popular for their reproducibility and ease of use.

Existing benchmarks mostly focus on household tasks. However, retail and logistics scenarios — such as shelf picking or order packing — remain underexplored. Dedicated benchmarks for these domains are needed to advance robotic capabilities in retail environments.

RoboBenchMart addresses limitations of prior works by providing code to generate diverse store layouts and robotic trajectories, enabling the training and benchmarking of robotic policies in retail environments.

Benchmark/ Dataset	Published in	Retail Domain	Scene Generation	Arrangement Generation	Release 3D Assets	Trajectories Generation	Tasks Diversity	Atomic Tasks	Composite Tasks
ALFRED	CVPR'20	✗	✓	✓	✓	✓	✓	✓	✓
RLBench	RA-L'19	✗	✗	✓	✓	✓	✓	✓	✓
RoboCasa	RSS'24	✗	✓	✓	✓	✓	✓	✓	✓
CALVIN	RA-L'22	✗	✗	✓	n/a	✓	✓	✓	✓
LIBERO	NeurIPS'23	✗	✓	✓	✓	✓	✓	✓	✓
VLABench	arXiv'24	✗	✓	✓	✓	✓	✓	✓	✓
BEHAVIOR-1K	CoRL'22	✗	✗	✓	✓	✓	✓	✗	✓
ManiSkill-HUB	ICLR'25	✗	✗	✓	✓	✓	✓	✓	✓
RP2K	arXiv'20	✓	✗	✗	✗	✗	✗	✗	✗
SKU110K	CVPR'19	✓	✗	✗	✗	✗	✗	✗	✗
StandardSim	ICIAP'22	✓	✗	✓	✗	✗	✗	✗	✗
IPA-3D1K	IROS'23	✓	✗	✓	✓	✗	✗	✗	✗
FetchBot	arXiv'25	✓	✗	✓	✗	✓	✗	✓	✗
RoboBenchMart		✓	✓	✓	✓	✓	✓	✓	✓

Comparing proposed robotics retail benchmark with other benchmarks and datasets

Store Plan Generator an open procedural pipeline for generating realistic and diverse store layouts and product arrangements. It enables scalable creation of retail environments for training and evaluating robotic policies.
Store Trajectories Sampler a pipeline that automatically collects trajectories for common retail tasks using motion planning and reinforcement learning methods. Moreover, we release a dataset of synthetic trajectories generated for the Fetch robot embodiment.
Store Robotics Benchmark the first open benchmark dedicated to evaluating robotic policies in retail environments. Using our benchmark, we demonstrate that current state-of-the-art models struggle to complete typical retail tasks.

We simulate dark-store environments as warehouse-style spaces filled with shelving and refrigeration units in diverse, randomized layouts. To support domain randomization, we vary wall, floor, and ceiling textures and use multiple fixture designs. Product items are then placed on shelves in realistic, slightly perturbed poses to mimic natural variability.

Fixture Arrangement. We seed a rectangular floor plan with pallets, boxes, and freezers. Rejection sampling guarantees collision-free initial placement. Next, we compute a smooth tensor field from store boundaries and fixture polygons. This field encodes local “flow” directions that naturally align aisles and corridors. Shelving is then placed in two passes—horizontal first, vertical second—following the local field direction. Every placement enforces clearance and minimum passage widths for navigation. A small probabilistic skip adds variety between scenes. The result is a clean, navigable layout that still looks different from run to run.

Product Arrangement. We use scene_synthesizer to detect shelf surfaces suitable for item placement. Items are placed on a grid with slight pose jitters for realism. The module supports vertical stacking to match common retail patterns. It can also leave front-edge gaps via a Poisson process to mimic partial depletion over time.

Assets & Textures. Our asset set includes 3 shelf models, 2 refrigerator models, and 370 product items across 21 categories. We also use 26 floor, 17 wall, and 15 ceiling textures for visual diversity. All assets are normalized for orientation and scale using retail reference dimensions. This keeps proportions realistic across scenes. To maintain performance with hundreds of items, we run an automatic mesh-simplification pipeline (QuadriFlow, Marching Cubes, and shape-aware approximations via the Blender API). From Pareto candidates, we pick meshes that minimize geometry error while maximizing triangle reduction. All curated and optimized assets are released publicly.

Examples of ceiling, wall, and floor textures used in our store generation pipeline, illustrating just a subset of possible variations

Asset simplification example 1 — Example of different geometry approximations for assets (original on the left). Numbers above indicate face count for each mesh

Asset simplification example 2 — Example of different geometry approximations for assets (original on the left). Numbers above indicate face count for each mesh

We generate training trajectories for dark-store tasks using motion planning. The resulting demos are suitable for imitation learning pre-training or fine-tuning. For each task, we define randomized anchor poses (start, intermediates, goal) to diversify demonstrations. The planner solves each segment between anchors in sequence. When the arm only moves, we attempt a fast screw motion (no obstacle awareness) and validate for collisions. If invalid, we fall back to RRT-Connect with obstacle checks. If both fail, we reset the scene and resample. When mobile base motion is needed, we use task-specific safety-aware heuristics to route the base and execute the arm locally. Result: feasible trajectories in ~60% of attempts across tasks.

We evaluate state-of-the-art generalist policies in retail settings, fine-tuned on trajectories from our sampler. The benchmark runs on ManiSkill3 for fast, realistic physics and ray-traced rendering. We use the Fetch mobile manipulator: differential-drive base, 7-DOF arm, and a prismatic torso for vertical reach. A parallel gripper enables reliable, generic grasping.

Testing Scenarios. We probe generalization along robot start pose, textures, store layouts, shelf arrangements, and item novelty. We report three tiers: In-Domain (start pose only), Unseen Scenes (start pose + textures + layouts), and Unseen Scenes & Items (plus OOD items from other tasks). Harder settings with completely unseen items/shelf layouts are supported but excluded—current policies fail even earlier tiers.

Tasks. Each task uses a text instruction with target item/fixture names. We check goal achievement and penalize unwanted collisions or scene disturbance.
Atomic tasks: pick to basket, pick from floor, from board to board, open fridge, close fridge.
Composite tasks: pick {N} items; pick from fridge (open → pick → close).

Generalist Baselines & Data. We fine-tune Octo and π₀ with imitation learning on trajectories from our sampler. To keep compute modest, we use 248 trajectories per (task, item, fixture)—2,480 demos total. To test cross-task generalization, we train on only 2–3 objects per task and keep shelves fully packed in train/test. Models are fine-tuned on atomic tasks; composites are executed as sequences of atomic instructions.

Evaluation results. We evaluate atomic and composite tasks using mean success rate per task, estimated from 100 trials per (task, item, fixture) triplet. The evaluation covers progressively harder settings: In-Domain trials, Unseen Scenes with new layouts and visual variation, and Unseen Scenes & Items with out-of-distribution objects. To explain low success rates, we also annotate failed episodes by the earliest primary failure in the manipulation pipeline, covering grounding, navigation, grasping, placement, and partial fridge-door articulation errors.

Model	Param. (#)	Testing scenario	Atomic Tasks					Composite Tasks
Model	Param. (#)	Testing scenario	Pick to basket	Pick from floor	From board to board	Open fridge	Close fridge	Pick 3 items	Pick from fridge
Octo	93M	In-Domain	21	1	21	27	45	0	0
		Unseen Scenes	2	1	1	23	26	0	0
		Unseen Scenes & Items	0	0	0	n/a	n/a	0	0
SmolVLA	450M	In-Domain	0	0	0	10	13	0	0
		Unseen Scenes	0	0	0	13	13	0	0
		Unseen Scenes & Items	0	0	0	n/a	n/a	0	0
π₀	3.3B	In-Domain	14	21	19	61	95	0	0
		Unseen Scenes	1	6	3	44	85	0	0
		Unseen Scenes & Items	0	1	0	n/a	n/a	0	0
π_0.5	3.3B	In-Domain	55	21	56	56	91	0	0
		Unseen Scenes	31	4	27	49	78	0	0
		Unseen Scenes & Items	5	3	21	n/a	n/a	0	0

Average success rates (%) of generalist VLA models on atomic and composite retail tasks across different testing scenarios. Higher values indicate better performance. n/a indicates that scenario is not applicable for the task.

The success-rate table shows that current generalist VLA models struggle even with basic retail tasks. SmolVLA performs poorly across all scenarios and succeeds only on the simplest fridge-opening and fridge-closing tasks, which may reflect its more limited pretraining data. Octo and $\pi_{0}$ reach moderate performance in the In-Domain setting, but their success rates degrade sharply on Unseen Scenes and collapse in Unseen Scenes & Items. $\pi_{0.5}$ performs best and is the only model with non-zero success in Unseen Scenes & Items, although its performance is still far from reliable. All models achieve zero success on composite tasks, suggesting limited ability to execute multi-step instructions and generalize across task stages.

	task	target	move	pregrasp	grasp	drop	displace	preplace	place-coord	place	partial
Octo	7	22	8	9	39	2	2	1	1	6	3
SmolVLA	4	29	16	32	17	0	0	0	0	0	2
π₀	0	23	1	9	43	5	1	2	4	5	7
π_0.5	0	20	1	2	50	4	7	0	2	9	5

Rates of failures (%) for each VLA model. For each model, the highest failure rate is shown in bold, the second highest is underlined, and the third highest is shown in italic.

The failure-rate table highlights two main bottlenecks. Octo, $\pi_{0}$, and $\pi_{0.5}$ fail predominantly at the grasp stage, indicating that precise shelf-level interaction remains difficult even when the correct region is reached. SmolVLA most often fails earlier, during pregrasp positioning. Also SmolVLA and Octo shows movement-related failures when approaching target fixtures. Target grounding is another common issue across models, showing that selecting the instructed product in cluttered shelves is still unreliable.

Overall, these results highlight three limitations of current generalist models: fragility to minor scene changes such as layouts, textures, and object placements; poor generalization from limited demonstrations to novel object-task combinations; and insufficient reliability for shelf-level grasping and compositional execution. This suggests that existing pretrained models may be insufficient for effective adaptation in retail environments without more targeted retail-specific pretraining.

BibTeX

@article{soshin2025robobenchmart,
      title={RoboBenchMart: Benchmarking Robots in Retail Environment},
      author={Soshin, Konstantin and Krapukhin, Alexander and Spiridonov, Andrei and Bukhtuev, Gregorii and Kuznetsov, Andrey and Shakhuro, Vlad and Shepelev, Denis},
      journal={arXiv preprint arXiv:2511.10276},
      year={2025}
    }

RoboBenchMart: Benchmarking Robots in Retail Environment

Abstract

Robotics Simulation for Retail

Contributions

Store Plan Generator

Store Trajectories Sampler

Store Robotics Benchmark

BibTeX