COLLAB-SIM is a research package for GPU-accelerated robot teleoperation with MPC and collection of teleop demonstrations in Virtual Reality (VR) in simulation. It runs VR with physics simulation and high quality rendering using NVIDIA's Isaac Sim and XR, enabling rendering of the simulated environment on the VR Headset at interactive speeds (30-35 FPS), while also streaming the VR HMD and controllers states (6-DOF poses and button states) back to the robot application.

This is a research preview release to enable researchers to explore uses of VR with robotics simulation and custom VR-based workflows for robot learning and human-robot interaction.

• Features
• Examples
• System Requirements
• Documentation
• Publications
• Citation
• License

The user teleops two Franka robots in Isaac Sim with left/right HTC controllers and Valve Index HMD, performing block stacking and object handover. Gif shown at 8X speed.

Example workflow for logging teleop demos and environment reset from VR controllers for starting a new demo. Block positions are lightly randomized at the start of each episode. Gif shown at 4X speed.

• Real-time VR Rendering: Interactive physics simulation at 30-35 FPS in VR
• Episodic Data Collection: Record and replay VR teleoperation demonstrations
• Custom Environments: Load USD scenes and configure robot base poses
• Flexible Callbacks: Implement custom workflows through VR controller button mappings

• MPC & IK Support: Model Predictive Control and Inverse Kinematics via CuRobo
• Delta Teleoperation: Intuitive end-effector relative motion control
• Gripper Control: VR controller button integration
• Multi-Robot: Support for single and dual-arm manipulation

This library prioritizes clarity and simplicity. We minimize abstractions to keep the codebase approachable for researchers new to VR robotics and Isaac Sim. Although the layout may not be fully optimized, this straightforward approach makes learning and following the steps easier.

• Meta Quest: Using ALVR on Ubuntu 22.04
• Valve Index: With HTC Vive controllers on Ubuntu 20.04/22.04
• HTC Vive: 2018 model on Ubuntu 20.04/22.04

Note: Other VR setups may work but are untested. We develop and test on Ubuntu due to its compatibility with robotics libraries, which are often Linux-based. Our focus is on configurations commonly used in robotics research and development.

• Installation Guide - Complete setup instructions for Isaac Sim, CuRobo, and VR
• Usage Guide - How to run examples and use the VR controllers

• collab_sim/demos/franka_mpc_example.py - Single Franka robot teleoperation with MPC
• collab_sim/demos/dual_franka_mpc_teleop.py - Bimanual dual-arm manipulation
• collab_sim/replay_states.py - Replay recorded VR demonstrations

If you use this work, please consider citing the following publications:

Fast Explicit-Input Assistance for Teleoperation in Clutter Nick Walker, Xuning Yang, Animesh Garg, Maya Cakmak, Dieter Fox, Claudia Pérez-D'Arpino IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024 Paper

Inference-Time Policy Steering through Human Interactions Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Pérez-D'Arpino, Dieter Fox, Julie Shah IEEE International Conference on Robotics and Automation (ICRA), 2025 Paper

If you find this work useful for your research and development, please cite as follows:

@misc{collab_sim_library,
    title={collab-sim library},
    author={Claudia P\'{e}rez-D'Arpino and Fabio Ramos and Dieter Fox},
    howpublished={\url{https://github.com/NVlabs/collab-sim}},
    year={2024}
}

See License file.