A developer has built a local 3D sandbox where a tiny robot with a big attitude builds whatever you ask for, entirely on your machine, with no cloud services involved. The project, llm-sandbox, splits interaction by speed: your hands handle movement and aiming through keyboard and mouse, while your voice handles intent like "build a tower" or "make it night." The model never sees the world. It only sees what you point at and what you say.
Why the model is blind on purpose
The sandbox runs Gemma 4 26B through llama.cpp, a mixture of experts model that delivers 26 billion parameters of knowledge at 4 billion parameter speed. But the model never receives a view of the 3D world. It receives only the text of what you said and a description of what you are pointing at. This constraint is deliberate. The gap between what you asked for and what the robot proudly builds is where the humor and the interesting failure modes live.
The robot talks back immediately, so the wait for the model to process your request feels like a conversation rather than a loading bar. A ghost outline appears where you aimed the instant you release the button, before the model has responded. The model then emits a short spoken acknowledgement in under a second, and the actual structure arrives a few seconds later. The design splits the feedback into two parts: immediate confirmation that your intent was heard, and delayed delivery of the result.
The model outputs volumes with stretch labels like tall, flat, and wide rather than individual cubes. The client expands those labels into blocks. This approach produces objects ten times larger with the same token budget, which matters when running on a single consumer GPU.
The full stack is local
The project runs four components, all local. The brain is Gemma 4 26B A4B via llama.cpp, running with strict JSON schema output so the world never receives malformed data. The ears are Whisper large-v3-turbo, fast enough that push-to-talk feels instant. The voice is Supertonic 3, a local text-to-speech engine pitched up to sound small and cheeky. The world is rendered with three.js in vanilla JavaScript with no bundler. The glue is a single Python file that runs a static server and proxies requests to the three services.
Everything is configurable through a single JavaScript file. Voice pitch, ghost size, token budgets, detail levels, language markers, key bindings, all live in config.js. Changing the robot's personality, speed, or size means changing a number. The code that implements the behavior does not need to be read or modified.
The project supports English and Spanish, detected per turn. Adding a language requires one config entry and two prompt files. The robot never repeats herself: words she used in the previous turn are forbidden in the next one. Without that constraint, the 26B model defaults to saying "amazing" forever.
What the controls actually feel like
Walking, looking, and aiming use keyboard and mouse, under 16 milliseconds of latency, with the model never involved. Push-to-talk uses right click: hold to talk, release to send. A ghost appears where you aimed while the model processes. Left click pins the aimed object as "this" or drags to move it. Middle click replicates the aimed object without calling the model. WASD moves, space jumps, shift sprints. F toggles flight and no-clip mode. Z and Y undo and redo.
The interaction model treats voice as a different input channel from direct manipulation, not as a replacement for it. You do not use voice to move, aim, or perform actions that have keyboard shortcuts. You use voice for things that have no key: build something, change the environment, add a capability. The latency of voice processing is acceptable for these actions because they do not require real-time response. It feels like casting a spell, not like waiting for a command to execute.
How it was built
The project was built by AI coding agents working in parallel, with a human directing by voice. Two files hold the project together: AGENTS.md, which contains the rules, the map, and the traps that already cost the developer time, and INTENT.md, which contains the reasoning behind every decision. The character, the rules she follows, and the output schema live in language prompt files, not in code. Changing who the robot is means editing a text file.
The test suite includes 564 unit tests that run without a browser or live services. Integration tests run against the three live services. The project requires Python 3, a browser, a GPU that fits a roughly 15 gigabyte quantized model, and the three local services running on their default ports. The services are not bundled. Anything that speaks the same contract works.
For developers interested in local AI interaction patterns, the split between fast direct manipulation and slower voice intent is a concrete design pattern worth studying. The model's blindness is a constraint that produces interesting behavior rather than a limitation that degrades the experience. The entire stack runs on one consumer GPU with no cloud dependency, which makes it a useful reference for building local AI applications that feel responsive despite model latency.