We built Navigate on openpilot in a video game
comma turned off Navigate on openpilot back in 2024. Last weekend at COMMA_HACK 7, my friend Kellen and I brought it back inside BeamNG.drive, and we took third.
Turns out the new driving model can already take a turn. It just has no idea when to, so most of what we built that weekend was us trying to tell it.
The setup
The hackathon was built around chestnut, comma's new external GPU for the comma four. It runs a 1B parameter driving model, about 30x bigger than what fits on the four by itself, and I wanted to know if all that extra model could actually get you somewhere, not just keep you in your lane.
We didn't have a car in San Diego, so we used a video game. BeamNG handles the graphics and the physics, and everything that decides what the car does runs on a real comma four with chestnut plugged in. I built the bridge that makes the game look like a real car to the comma, and Kellen did the models.
openpilot only gets the camera, the car's CAN data, GPS, and a map, same as it would in a real car. The game knows a lot more than that, like the actual color of every light, but we only used that afterward to check whether the car had broken any laws.
The blinker test
The old Navigate on openpilot showed the driving model a picture of the map. The new model doesn't take a map at all. The only way to tell it about a turn is "desire," which is the same signal your blinker sends: turn left, turn right, keep left, keep right.
That's what we tested first. We rolled up to an intersection, sent "turn right" and nothing else, and the car took the turn on its own. At the same intersection with desire turned off, it drove straight through and never even started to turn. The turn was already in the model, we didn't have to teach it that part at all.
Telling it when
We wrote navd to do the telling. It takes a map, a destination, and GPS, figures out the route, and sends the right desire at the right time. Getting the timing right ate most of the weekend.
Our first version said "keep left" 150 m before every fork, which is a long time to be told to keep left when there's nothing to keep left of, and three drives ended in the dirt because of it. Now it waits until about 2.5 seconds before the fork. It also shuts up if the camera can see there's no lane on that side, which we added after the car, already sitting in the leftmost lane, got told to keep left anyway and left the road at about 65 mph.
Left turns have their own problem. You need to be in the left lane, and you really don't want to change lanes into oncoming traffic. The camera can see lane lines, but it can't tell you which way the next lane is going, so navd cheats and checks the paint. It'll cross a white line and it won't cross a yellow one.
Cloverleafs were cursed. The route folds back on itself, so at one point the lane going the wrong direction was 2 meters away and the correct one was 58 meters away, and navd kept picking the close one. It rerouted 66 times in one drive, and the trip got longer with every reroute. Now it checks direction first and only uses distance as a tiebreaker, which in hindsight should have been obvious.
Okay, this part I'm not proud of. For the turns themselves, navd draws its own path through the intersection and blends it into the steering. At full strength that means a hand-written path is steering the car, not the model, which defeats the whole point of the blinker test. We did it because the demo was on Sunday, and it's the first thing I want to delete.
Red lights
The big model doesn't care about BeamNG's traffic lights at all. We parked it in front of one for two full red-green cycles and nothing changed. Part of that is resolution, by the time the image gets to the model a light 60 m away is about three pixels wide.
Kellen ended up training lightd, a small second network that reads the camera at full resolution and only looks up where traffic lights actually are. The training data was free since the game can force any light red, yellow, or green on command, and on the map it trained on it caught almost every light. Then we tried a map it had never seen and it ignored a huge chunk of them. Every light on the training map had a yellow or grey housing, so it had decided a black housing meant it was looking at the back of the light. We painted the training lights random colors and added a class for dark housings.
It also saw a yellow road sign once, decided it was a yellow light, and held the car for the entire trip. Real yellows turn red in a few seconds, so now anything that stays yellow for 10 gets ignored.
It ran a red once too. We'd slowed lightd down to save compute, and the code that confirms a stop was counting frames instead of seconds, so half the frame rate meant twice as long to confirm. The light dropped out for one frame right before the line and the car took that as permission to go. We found two more timing bugs on Saturday, both in my bridge code.
The bridge
The comma has no idea it's in a video game, and getting it there took most of my weekend.
Frames first. A Lua mod I wrote spawns a second, off-screen camera inside BeamNG, mounted 1.3 m up where a comma would sit on the windshield, and saves every frame as a JPEG. The player's view, the HUD, and anything we drew on the road never reach the model. My first version just screen-captured the game window, which was way cheaper (the game ran at 86 fps instead of 16), but then the model saw wherever the player's camera happened to be pointing. One render covers both of the comma's cameras, since the middle of the wide camera is exactly the road camera, so the relay crops it out instead of rendering twice.
From there frames go from the gaming PC over a USB4 cable to a Mac, then over WiFi to the comma. On the comma I block the real camerad, sensord, and pandad, and one Python process takes over for all of them. It decodes each JPEG straight into the same NV12 buffers the real camera sensor fills and publishes them over VisionIPC under camerad's name, so modeld can't tell the difference. It also fakes a Tesla Model 3, packing real CAN messages with opendbc's own DBC file. openpilot's actual Tesla port reads them, which means the real Tesla steering limits sit between the model and the wheel. The IMU comes from the game's physics at 100 Hz and GPS comes from converting the game's coordinates to latitude and longitude. There's no panda anywhere.
WiFi broke first. My relay used a request-and-response protocol, and over WiFi only 2 to 5 frame pairs a second made it to the comma while the game was making 14. Moving the adapter onto the Mac, so only one-way streams cross WiFi, fixed that, and then everything else broke in order. Frames showed up in clumps and modeld started skipping frame ids, so I paced them out. The WiFi topped out around 65 Mbit/s, so I cropped the wide image down to the part the model actually uses, which took it from over 100 KB a frame to about 30. JPEG conversion on the comma went from 68 ms a frame to 23. And every time modeld loaded the big model it starved my bridge long enough that openpilot went five seconds without hearing from the panda, which doesn't exist, decided the car was off, and dropped offroad, over and over. Realtime scheduling priority fixed that one.
Both of Saturday's timing bugs were in here too. The comma's clock ran 40 ms ahead of the Mac, and my frame-age math only ever corrected in one direction, so every frame looked 40 ms older than it was. That turned my 300 ms stale-frame cutoff into a 260 ms one, and frames in between got dropped without anything complaining. The other was a status field that reported the device's uptime as the frame's age before the first frame arrived. At one point it told me a frame was 97 minutes old.
One timing problem is openpilot's, not mine. modeld assumes every frame is 50 ms old. While driving, ours had a median age of 119 ms and never came in under 50, and from the camera to a steering command took about 200 ms. So the car was always driving on a picture that was older than it thought. We have a patch that measures frame age instead of assuming it, but it never made it onto the car.
My favorite bug of the weekend happened at a red light on a hill. The car started rolling backward while openpilot thought it was going forward, because the game was sending speed instead of velocity, and speed doesn't know which direction you're going. openpilot saw a car creeping forward through a red and gently tapped the brakes... while it rolled backwards down the hill.
The demo
By Sunday it was actually working. In one drive it made 20 turns and lane choices in a row without a single navigation mistake, and it got itself back on route every time it had to reroute. On a separate lights run it stopped at every red and went on every green, and when we checked it against the game afterward it hadn't run any of them. It even stopped at a stop sign in the middle of a route and kept going. Our demo loop was about 7.6 km with lights, stop signs, and two freeway exits.
Kellen also trained mapnet, a tiny network that looks at a picture of the local map and says where to go next. It's the same idea as comma's old nav model, just a lot smaller. It learned on real streets in Seattle and California and worked in the game on the first try, but it didn't make the demo.
TurnPilot
The hackathon's over, but we're keeping this going. We're calling our fork TurnPilot. Right now navd steers the turns and the model mostly rides along, which is backwards. Where I'd like to get is giving the model more of the route than a blinker, maybe something like mapnet's output, so navd can stop steering. No promises on when.
The other big step is getting out of the game and into a real car. On the highway openpilot barely moves the wheel, but a city turn needs the car to stop with nobody in front of it, creep at walking pace, and crank the wheel most of a full turn. A lot of cars can't do all of that. Fords can't ask for a turn tighter than about a 50 m radius, and a normal corner is closer to 8 m. Kellen went through all 341 cars openpilot officially supports, and on paper 165 of them pass. We're only really confident about 8 though, mostly Teslas, because they steer by angle instead of torque. For the other 157, someone has to drive a real corner in each one before we know.
Subscribed. The next post will come to your inbox.