14 Robot Skills in 5 Days on a Single GPU: Queued RL Training for a Bipedal Robot Duck
MicroDuck, a lightweight bipedal robot, achieved a full library of 14 skills—including walking, skating, and kicking—within five days on a single GPU. The team used MuJoCo Warp, PPO, and a series of engineering tricks such as serial task queues, dynamic environment scaling, and NaN‑safe patches to…
MicroDuck, a bipedal robot duck equipped with 15 XL330 servos and a Radxa ZERO 3W board, has just finished training a complete set of 14 skills on a single GPU in under five days. The skills range from basic locomotion like walking and skating to more complex tasks such as kicking a ball and picking up objects. The project demonstrates that sophisticated multi‑skill learning can be achieved with modest hardware when coupled with careful engineering and reward design.
Training Stack and Hardware
The simulation environment was built on mjlab, a MuJoCo wrapper that leverages the Warp GPU‑parallel backend. Reinforcement learning was carried out with Proximal Policy Optimization (PPO) from the rsl_rl library. The team ran 4,096 parallel environments on flat terrain and reduced this to 2,048 for rough terrain to avoid out‑of‑memory (OOM) errors. All training was performed on a single GPU, with a 50 Hz control loop and a serial task queue to prevent GPU contention.
Skill Matrix and Results
- Velocity‑Flat (walking) – 25k iterations
- Velocity‑Rough (rough‑terrain walking) – 25k iterations
- Velocity‑Rollers (skating) – 25k iterations
- Velocity‑Swizzle – 25k iterations
- VelStand (walking + recovery) – 10k iterations
- StandUp – 7.5k iterations
- GroundPick – 10k iterations
- BallKick – 5k iterations
- Roulade (rolling) – 7.5k iterations
- SitStand – 7.5k iterations
Each skill was validated with a 300‑frame rollout GIF, providing a clear visual confirmation of performance. The team noted that a smooth loss curve does not guarantee a good gait; the GIFs are the definitive acceptance test.
Engineering Tricks for Long Runs
Three key techniques kept the training pipeline robust:
- Serial queue with .done markers – Tasks are executed one after another, and a marker file signals completion, allowing unattended runs.
- OOM downgrade‑and‑retry – Rough‑terrain tasks initially crashed due to VRAM limits; reducing the environment count to 2,048 resolved the issue.
- NaN‑safe patches – NaN errors were caught by inserting guards on rewards and advantages, with a flag that triggers episode termination when NaNs appear.
Reward Design as an Economics Problem
Each skill employed a distinct reward philosophy, all guided by a simple rule: reward terms should be small enough (≈10) that regularization does not dominate learning. The team reused proven reward recipes from a baseline velocity task and added only the necessary task‑specific layer. Examples include:
- Velocity tasks – A slight forward lean was penalized minimally to encourage natural gait.
- StandUp – A multi‑layer height reward combined dense and sharp gradients to help the robot escape the sitting basin.
- Roulade – The reward encouraged rapid accumulation of 2π rotation while penalizing overspeed and enforcing ground contact.
- BallKick – The actor was blind to the ball, while the critic could see it, providing asymmetric learning that mimics real‑world constraints.
- GroundPick – Instead of specifying a pose, the reward used task‑space forces to keep the mouth near the ground.
- VelStand – Recovery was only rewarded when a fall actually occurred, preventing the policy from exploiting the recovery layer during normal walking.
Key Takeaways
- Serial queues and .done markers make single‑GPU training reliable.
- OOM and NaN failures require different diagnostics: OOMs show up quickly in logs, NaNs appear later and need wandb curve analysis.
- Rollout GIFs are the most trustworthy way to verify skill performance.
- VRAM budgeting suggests using no more than 70% of environments for rough terrain.
- Reward terms should be treated like taxes; too high a penalty can kill exploration.
What’s Next?
The policy library has been exported as ONNX models, ready for deployment on the physical MicroDuck. Future work will involve testing these policies in real‑world scenarios and refining the reward structures further. The team remains open to collaboration and discussion about the methods used.
Why it matters
Demonstrating that a complex multi‑skill robot can be trained on a single GPU in days lowers the barrier to entry for robotics research and accelerates the deployment of autonomous systems.
Key points
- Single GPU trained 14 skills in 5 days
- Used MuJoCo Warp, PPO, and serial task queues
- Implemented OOM downgrade and NaN‑safe patches
- Reward design treated as economics to avoid over‑penalizing
- Rollout GIFs are the definitive acceptance test
Frequently asked questions
How many environments were used during training?
The team ran 4,096 environments on flat terrain and reduced to 2,048 for rough terrain to avoid memory issues.
What is the purpose of the .done marker files?
They signal task completion, enabling an unattended serial queue that prevents GPU contention.
Why were NaN‑safe patches necessary?
NaN errors can appear after many iterations; the patches guard rewards and advantages to terminate episodes when NaNs are detected.
