Dexterity Interface

Setup

1. Install Requirements

You will need:

• For the simulation/interface:

  • Ubuntu machine
  • Docker Engine.
  • A live server. For VScode, we recommend Live Server Extension.

• For Running on the Panda:

  • Above requirements.
  • Franka Emika Panda Robotic arm with a Force Torque Sensor. Reference panda-primitives-control for the specifics.

2. Setup LLM configs

1. First, you need to create a .env file in this folder with the OpenAI credentials. It should be in this format:

   OPENAI_API=YOUR_API_KEY_HERE

3. Set up the repository and run the container

1. Bring in the submodules:

   git submodule update --init --recursive

2. Now build the container image and start the container. Make sure you are in this root directory. These commands mount on the current directory as the containers file system so any changes you make to the files on your host machine will be mirrored in the container. These commands also allow the containers display to be forwarded to your host machine so that you can see it.

   sudo docker build -t llm-control .
   
   sudo docker run --rm -it --privileged --cap-add=SYS_NICE --device=/dev/input/event* --env DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v $(pwd):/workspace --net=host llm-control

4. Compile ros packages

This is a bit over-complicated due to having multiple workspaces. Turning some of these workspaces into packages is a future TODO item.

cd panda-primitives
catkin build authoring
source devel/setup.bash

cd ../dexterity-interface
catkin build
source devel/setup.bash

Running

1. Launch the Backend. Make sure you are in the dexterity-interface/ directory when running this command:

   roslaunch interface backend.launch

   Note: If you run into an issue with catkin not being able to find the authoring package, please run catkin clean -y in the dexterity-interface/ and the panda-primitives/ directories and then redo Step 4 in the setup. After that, this command should work.

2. Launch a live server for frontend/index.html. If you are using VScode, you can do that by selecting that file to open it, and in the lower right of VSCode click "Go Live". This should launch the interface in your browser.

Troubleshooting

# To view TF frames
rosrun tf2_tools view_frames.py

Resources

• https://github.com/cruise-automation/webviz
• https://github.com/osrf/rvizweb
• https://robotwebtools.github.io/
• https://github.com/Mechazo11/interactive_marker_proxy_noetic
• https://github.com/ros-visualization/visualization_tutorials/tree/noetic-devel/interactive_marker_tutorials
• http://wiki.ros.org/roslibjs/Tutorials/BasicRosFunctionality
• https://robotwebtools.github.io/ros3djs/
• https://docs.ros.org/en/noetic/api/visualization_msgs/html/msg/InteractiveMarkerControl.html