Robo, the Makerspace Greeter

01

What Robo is

Robo is a table robot built from a ROBOTIS OpenMANIPULATOR-X arm and an NVIDIA Jetson Orin NX 16GB running ROS 2 Humble. The goal: it sees people with a camera, greets them with gestures and voice, listens to commands, and picks up small objects like a pencil.

It lives at the Ingram Hall Makerspace at Texas State University, and the control panel is styled in Texas State maroon and gold. Everything (vision, hearing, language model and voice) runs on the Jetson itself.

02

What it can do

• Gestures: wave, nod, shake, bow and look around, plus your own moves: record them in simulation and replay them on the real arm.
• Sees people and objects: YOLO and YOLO-World on the Jetson GPU run live from a Logitech BRIO. Robo waves automatically when someone steps into view, and greets each person once.
• Listens: Whisper on the GPU turns speech into text, for voice commands like “wave” or “go home”.
• Talks: a local Llama 3.2 3B (via Ollama) decides what to say and which gesture to use, and Piper speaks through a USB speaker. Robo only talks about robotics, itself, Texas State and the Makerspace, and politely declines anything else.
• Control panel: one window with gestures, joints, gripper, recording, live video with detection boxes, talking with Robo and the games.

03

The arcade games

A camera on a stand tracks the player’s hand with MediaPipe, and Robo’s arm follows it, so anyone can try robotics just by raising a hand.

1. Puppet Master

   Your hand controls Robo’s gripper and a fist whacks. Target stickers light up on the table; whack them before the timer runs out, avoid the red ones, and build combos. A 60-second round, with ranks from C to S and a leaderboard.

2. Ball Carry

   Carry the balls across the table with the claw; the fastest time wins.

3. Chocolate Catcher

   A claw machine with Robo as the claw. Move your hand to steer, make a fist to grab, and open your hand over the basket to win the chocolate. Five lives shown as hearts, a 10-second carry timer, and a 2-minute game, with sound effects generated in code.

The safety rules are in the code: the arm only moves inside the play area, never faster than 1.5 rad/s, never below the touch height, and it holds still when it loses sight of the hand.

04

Engineering notes

The control stack is layered: gesture and game code send joint angles over a ROS 2 action, an arm controller turns them into a smooth path, the Dynamixel hardware interface sends servo commands over USB to the OpenCR board, and the servos report their angles back as joint states.

• A shared GPU is a budget: YOLO checking 30 frames a second slowed the LLM and Whisper about 3.5× (11.5 tok/s alone, 3.3 with everything running). Pausing detection while Robo talks cut the wait for first words from 15.8 s to 6.6 s.
• Faster turn-taking: Whisper starts transcribing the moment you pause and discards the work if you keep talking; Robo decides you’ve finished after 0.8 s of quiet; and every answer begins with a very short sentence so speech starts almost as soon as the LLM does.
• Calmer servos: the default base-servo gains made the arm jitter about ±0.3° while holding a pose. Gentler P/I/D settings, sent at every launch, fixed it without changing the servo permanently.
• Reproducible setup: the README walks through the whole Jetson setup, including pinned Python versions so ROS 2 Humble, NumPy and OpenCV keep working together.

05

Where it stands

• Done: Arm control, gestures, recorded moves, gripper and the button panel
• Done: Camera, live YOLO / YOLO-World detection and automatic greeting
• Done: Speech to text, text to speech and a local LLM
• Not yet: Voice commands and talking with the arm attached
• Not yet: One-command launch with the full voice-enabled panel
• Not yet: Saying hello while it greets, and picking objects up

Unchecked items are written but still being tested on the real arm.