[Question] How to perform teleoperation with direct environment?

[IrvingF7]

Question

After reading the documentation and teleoperation source code like the manipulation-lift environment, I think I have a rough idea of how to convert any managed environment to be teleoperable. Basically, you can assign the action manager actions.arm_action to a DifferentialInverseKinematicsActionCfg and the teleop script can handle the interfacing between the action manager and the teleop device.

However, I am not sure how to do something similar with direct environments. I would really like to convert some Factory and Automate environments into teleoperable environments. Direct environments don't have an action manager, so while I suspect the teleop script can still be used (since it only calls env.step(), but not the action manager itself), I am not sure how to introduce such a "action-manager-like" component into it and there is no example like that.

I will explore this on my own (and hopefully contribute a PR if possible), but I would also greatly appreciate it if someone could provide some pointers. Thanks in advance

[RandomOakForest]

Thank you for posting this question. It is a great conversation starter for our Discussions section. I'll move the post there for follow up. In the meantime, here is a summary I'm still reviewing, but you may consider it to get you started.

Key Differences Between Manager-Based and Direct Workflows

• Manager-based environments use an ActionManager to process actions (¹²), enabling easy teleoperation integration through predefined action terms.
• Direct environments bypass managers and interact directly with physics, requiring manual handling of teleoperation inputs (³⁴).

Implementation for Direct Workflow Teleoperation

1. Create a Teleoperation Wrapper Class
   Intercept actions from teleop devices and map them to environment actions:

class DirectTeleopWrapper:
    def __init__(self, env, teleop_device):
        self.env = env
        self.teleop_device = teleop_device  # e.g., SpaceMouse
        self.ik_solver = DifferentialInverseKinematics()  # Add IK solver
    
    def step(self, actions=None):
        if actions is None:
            device_command = self.teleop_device.get_command()  # SE(2)/SE(3) input
            actions = self._convert_to_direct_action(device_command)
        return self.env.step(actions)
    
    def _convert_to_direct_action(self, command):
        # Convert teleop command to joint-space actions
        joint_positions = self.ik_solver.solve(command)
        return joint_positions  # Direct env expects raw joint commands

2. Modify Environment Initialization
   Replace the standard environment with the wrapped version:

# Original direct env
env = gym.make("Isaac-Factory-Direct-v0") 

# Teleoperable version
teleop_env = DirectTeleopWrapper(env, teleop_device="spacemouse")

3. Handle Action Space Compatibility
   Ensure teleop outputs match the environment's action space:
   • For joint position control: Scale IK outputs to [-1, 1] range.
   • For velocity control: Convert pose deltas to joint velocities.
   • Use clipping/normalization to match action space bounds.
4. Add Teleoperation-Specific Parameters
   Extend environment configuration to include:

@configclass
class DirectEnvTeleopCfg:
    teleop_device: str = "keyboard"  # or "spacemouse"
    max_lin_vel: float = 0.5  # Tune for task
    max_rot_vel: float = 1.0

Critical Implementation Details

• Inverse Kinematics Requirement
  Direct environments lack built-in IK, so integrate a solver like:

from isaaclab.controllers import DifferentialInverseKinematics
self.ik_solver = DifferentialInverseKinematics(robot_cfg, gravity_compensation=True)

• State Tracking
  Maintain robot state internally since direct envs don't expose managers:

self.current_joint_positions = env.get_joint_positions()

• Command Processing
  Convert teleop device outputs to environment-compatible actions:

def _process_spacemouse(self):
    raw = self.device.get_state()
    return [
        raw.twist[^0] * self.max_lin_vel,  # Linear velocity scaling
        raw.twist[^5] * self.max_rot_vel   # Angular velocity scaling
    ]

Usage Example

./isaaclab.sh -p custom_teleop_direct.py \
    --task Isaac-Factory-Assembly-Direct-v0 \
    --teleop_device spacemouse

Key Advantages

1. Minimal Env Modifications
   Avoids altering core environment logic by using a wrapper.
2. Reusability
   Same wrapper works for any direct environment with joint control.
3. Performance
   Maintains direct workflow's low-latency advantages while adding teleoperation.

This approach effectively bridges teleoperation capabilities into Isaac Lab's direct workflows, enabling real-time human control for tasks like factory automation and robotic assembly³⁴. The wrapper handles command conversion while preserving the direct environment's performance characteristics.

References

Footnotes

1. https://isaac-sim.github.io/IsaacLab/main/source/api/lab/isaaclab.managers.html ↩

2. https://isaac-sim.github.io/IsaacLab/main/source/api/lab/isaaclab.envs.html ↩

3. https://forums.developer.nvidia.com/t/isaac-lab-camera-sensor-with-manager-based-environment-or-direct-workflow/300073 ↩ ↩²

4. https://forums.developer.nvidia.com/t/error-when-creating-an-original-direct-workflow-rl-environment-in-isaac-lab/301047 ↩ ↩²