Building a 15 kg Humanoid Robot That Understands Gestures, Speech, and Objects for Under $10,000
Humanoid robots that interact naturally with people in real-world environments are expensive. Unitree's G1 and R1 cost tens of thousands of dollars. Boston Dynamics' Atlas is a research institution's budget line. Tesla Optimus and Engineered Arts' Ameca are closed-source and prohibitively priced for university labs. The result is a gap between high-performance platforms that academic researchers cannot afford and low-cost alternatives that are too weak or too simple for sophisticated human-robot interaction experiments. A team from Vietnam National University and JAIST presents a humanoid robot prototype that tries to bridge this gap: a 150 cm, 15 kg platform with a 12-DOF dual-arm mechanism, an expressive LCD face, and three integrated AI modules for gesture recognition, object detection with 3D localization, and voice-command processing through LLM-based semantic parsing.
Mechanical Design: Aluminum Frame, 3D-Printed Joints, Custom PCB
The robot uses a hybrid construction. Lightweight aluminum bars form the structural frame, providing durability and load-bearing capacity for the arms and body. Custom 3D-printed parts form the joints and brackets, enabling rapid fabrication and low-cost replacement. The result is a platform that weighs only 15 kg while standing approximately 150 cm tall, close to the scale of an average human torso. This human-scale design means gestures like handshakes, waves, and object handovers occur at natural heights, making interaction experiments more realistic.
Each arm has 6 revolute joints actuated by serial-link servos. The servos are allocated by joint load: high-torque DS5160 units (65.0 kg-cm) drive the shoulders, FT5330M (35.5 kg-cm) drives the elbows, and lighter TD-8120MG and MG996R units control the wrist and end-effector. The maximum payload is approximately 600 g. The arm kinematics follow standard Denavit-Hartenberg conventions for forward and inverse kinematics computation. The head has 2 DOFs (yaw and pitch) for directional attention, and a 7-inch LCD screen displays facial animations corresponding to the robot's emotional state.
All electronics are coordinated through a custom-designed 12x12 cm controller board. A dual-microcontroller architecture runs the system: an Arduino Mega 2560 controls the 12 DOF of the dual arms (chosen for its large I/O pin count), while an Arduino Uno manages the head module and sensors. They communicate via I2C. The board also integrates Bluetooth for wireless teleoperation. A ZED 2 stereo camera mounted above the screen provides RGB and depth perception. An NVIDIA Jetson AGX Xavier (8-core ARM CPU, 512-core Volta GPU, 32 GB RAM) serves as the main processing unit, running ROS for system coordination and handling all AI tasks. A 36V battery provides power, stepped down to 19V for the Jetson and 5V for the controller board and servos.
Three AI Modules: Vision, Gesture, and Language
The software architecture has three modular AI components that collectively enable perception, understanding, and response.
Object detection uses the ZED 2 stereo camera to acquire RGB and depth data. YOLOv9 identifies object classes and bounding boxes in pixel space. These pixel coordinates are projected into 3D world coordinates using the stereo camera's depth map. Intrinsic calibration with a chessboard pattern corrects for lens distortion. The resulting (X, Y, Z) coordinates feed into inverse kinematics to generate arm control commands for pick-and-place operations.
Gesture recognition uses MediaPipe Pose for real-time 2D keypoint extraction from the human body. An LSTM network classifies sequences of keypoints into predefined gesture categories: waving, handshaking, and love gestures. The model was trained on a custom dataset captured under varied lighting and distance conditions. When a gesture is detected, the robot responds with corresponding arm motions and facial expressions on the LCD screen.
Voice command processing captures audio through a USB microphone and transcribes it using the SpeechRecognition library. The transcribed text is passed to an LLM (Gemini API) through a structured prompt that outputs an action-object-location triplet in JSON format. For the command "Take a cup and put it on the plate," the LLM returns an action of "pick," an object of "cup," and a destination of "plate." If the objects and locations are detected by the vision module, their 3D coordinates are retrieved and translated into robot commands like PICK(x1, y1, z1), MOVE(x2, y2, z2). The structured JSON output reduces token generation, keeping response time to about 2 seconds on average.
Experimental Results: 96% Gesture, 92% Speech, 1.83 cm Error
Two experimental scenarios validate the system. In the first, the robot faces a user from approximately 1.5 meters. The LSTM classifier achieves 96% accuracy across the three predefined gestures, with an average response latency of 1.2 seconds from gesture completion to robot action initiation. The robot waves back with a smiling face, performs a handshake, or responds to the love gesture with corresponding arm motions and facial expressions.
In the second scenario, 300 randomly issued spoken instructions evaluate the voice pipeline. Speech recognition achieves 92% transcription accuracy. The LLM-based intent parser achieves 96% semantic parsing accuracy. YOLO-based object detection maintains over 90% accuracy across varying lighting conditions. The robot successfully executes pick-and-place tasks with these parsed commands.
Positioning accuracy is measured across 10 trials for each arm. The left arm averages 1.30 cm error, the right arm averages 1.28 cm. When YOLO detection and 3D transformation are added to the pipeline, the integrated system averages 1.83 cm error. The overall pick-and-place success rate is approximately 90%. The variability is notable: one trial reaches 3.82 cm error while another hits 0.41 cm, suggesting that detection noise and calibration consistency dominate the error budget.
Trade-offs and Limitations
The platform makes deliberate trade-offs. The 600 g payload limits the robot to lightweight objects. The 12-DOF dual-arm design provides sufficient dexterity for basic manipulation but not for complex bimanual tasks. The LCD face displays emotions but does not move physically, which reduces the expressiveness compared to platforms like Ameca that use mechanical facial actuators.
The biggest limitation is the reliance on external LLM APIs. The Gemini API introduces latency, requires internet connectivity, and creates a dependency on a third-party service. This is a practical constraint for real-world deployment, particularly in environments with unreliable connectivity. The authors identify integrating lightweight, on-device language models as a key direction for future work.
The evaluation is also limited in scope. The gesture classifier handles only three gestures under controlled indoor lighting with minimal occlusion. The voice pipeline tests 300 commands, which is small compared to the diversity of natural language in real interaction. The positioning experiments use a fixed base, so the robot does not walk or navigate. These are honest limitations for a prototype, but they frame the system as a research testbed rather than a deployable platform.
What This Means for HRI Researchers
The value of this work is not in pushing the state of the art on any individual metric. It is in assembling a complete, functional humanoid platform that integrates perception, language understanding, and manipulation at a price point accessible to university labs. The 3D-printed construction and off-the-shelf servos keep costs low. The modular software architecture means individual AI modules can be swapped or upgraded independently. The Jetson AGX Xavier provides enough GPU power for real-time inference without requiring external compute infrastructure.
For researchers building HRI experiments, this offers a reference architecture: how to wire a stereo camera to YOLO for object detection, how to chain MediaPipe Pose to an LSTM for gesture classification, how to structure LLM prompts for action-object-location extraction, and how to connect all three through ROS to a dual-arm manipulator. The code and design files are not explicitly released as open-source in this paper, but the architecture is described in enough detail to reproduce.
The 96% gesture recognition and 92% speech recognition numbers, combined with sub-2 cm positioning error, suggest the platform is functional enough for controlled HRI studies. The 1.2-second gesture response latency and 2-second voice command latency are acceptable for laboratory experiments but would need improvement for natural conversation-speed interaction. Future work on on-device language models would address both the latency and the connectivity dependency.