Oil painting of a sunny downtown San Diego intersection seen from the driver's seat: trees along both sides, a mast arm reaching over the road, and red traffic lights, each outlined with a red detection box.

Getting lightd out of the video game

By Chase / @explodingcb ·

In the last post, lightd was the little network that kept our car from running reds in BeamNG. It worked because the game cheated for us. BeamNG knows the color of every light at every moment, so we had perfect labels for free, as many as we wanted.

Real roads don't do that. Nobody hands you the color of every light in Seattle, and the goal for TurnPilot is supervised drives on real Seattle roads by the end of the year. So this week was about getting lightd out of the video game and pointed at an actual comma camera. Kellen had already built the training code for real footage, and my job was the data, so I spent the week feeding it, labeling for it, and watching it train on my RTX 5080.

We ended up with lightd v4. It reads close daytime lights well, it has never once called a red light green, and the weights are up on Hugging Face. It's not done though, and most of this post is about that.

No game to copy off of

Without the game we needed real labeled lights, which means public datasets. We used LISA (driving clips from San Diego), Bosch's traffic light set, and S2TLD. Every image gets rescaled so the lights come out the same size they would be on a comma four's road camera, and then cut into crops so the model sees a lot of different lights per batch.

Grid of augmented training crops of street scenes with colored boxes around traffic lights.
Training crops from LISA, rescaled so the lights are the size they'd be on a comma four. Boxes are colored by the labeled color, and the soft rectangles are rare lights (yellows, arrows) pasted in so the model sees more of them.

We trained two models. The teacher is big and slow, and that's fine because it never leaves my PC. The student is the one we would actually put in a car, an EfficientNet-B0 with about 3.7M parameters, and it learns by copying the teacher's answers.

The first teacher runs were already weird. They got better for 6 to 10 epochs and then just started getting worse, even though training loss kept going down the whole time. That's memorizing. LISA's whole training set is 13 video clips and Bosch's is 5 recording sessions, so of course it is, the model sees the same handful of intersections over and over.

Line chart of validation AP by epoch for two teacher runs, peaking at epoch 6 and epoch 10 and then declining.
Both teacher runs peaked early and then got worse on held-out clips while their training loss kept falling.

LISA just didn't label the lights

So we went looking at its mistakes. These are the teacher's most confident "false positives" on LISA, the stuff it said was a light that the dataset says isn't.

Basically all of them are real, lit traffic lights. LISA just never labeled them. A lot of these are the same light across several frames of one clip, and the model is right about every single one. The only real mistakes in there are a few neon shop signs.

Contact sheet of crops of lit traffic lights with red boxes and scores between 0.60 and 0.81.
The teacher's most confident "false positives" on LISA. Nearly all of them are lit lights LISA never labeled.

That's a problem for training too, not just for scoring. If a real light is not labeled, the model gets punished for finding it, and after enough of that it learns to be kind of scared of lights. Which is exactly what it was doing: the lights it "missed" at strict cutoffs were big, obvious, lit lights. It saw them, it just wasn't willing to commit.

The fix was to let the teacher flag its own confident detections that match no label and turn those into "ignore" regions, where the model gets no reward or punishment either way. When we checked, about 9 in 10 of them were real lights.

Contact sheet of crops, mostly green and red traffic lights at night, with yellow boxes.
Detections that became "ignore" regions on LISA. About 9 in 10 are real lights.

Bosch has the opposite problem. It doesn't label signal heads that face sideways or away from the camera, and we tried ignoring those too. That made the model start firing on the backs of lights, so we put it back. For driving that's fine anyway, since a light facing away from you isn't your light.

The comma camera

The public sets are all shot on somebody else's camera. What actually matters is how lightd does on a comma, and there is exactly one openly licensed pile of real comma-camera frames: comma10k. So we hand-labeled 240 frames from it, 445 lights total, all from cars the model never trained on. That's the scorecard for the rest of this post.

We also let the teacher label the rest of comma10k so the student could train on real comma frames. Reds and greens came out great.

Large grid of small crops of red traffic lights from comma camera frames.
The teacher labeling comma10k: everything it called red. Nearly all correct.

Yellows did not. About 69% of what it called yellow on comma10k were sodium streetlights and yellow diamond road signs, so we threw all 198 of them out. Later Kellen went through comma frames by hand and found 71 real yellows, which went into v4.

Grid of crops that are mostly orange streetlight glare, with only a few real yellow signals.
Everything the teacher called yellow on comma10k. Mostly sodium streetlight glare and yellow diamond signs, so all 198 went in the trash.

(We also tried getting AI vision models to check labels for us. The local one agreed with us 77% of the time, which for traffic lights is a pretty bad number, and the free Gemini key rate-limited us, so it was humans or nothing.)

Okay, this one's on me. For one entire overnight run the comma frames weren't even in the training mix, because a config key didn't match and comma10k got a sampling weight of zero and nothing complained about it. Student v2 trained all night and never saw a single comma image.

The other thing the comma camera does is blow up close lights. When you're right under a red, the lamp turns into a huge glowing disc, white in the middle with a red halo, and the public datasets barely have any of that. So we added a fake "lamp bloom" to training that blows random lights out the same way. It even fades the halo in daylight, since a colored glow only really shows up against a dark background.

Here's what the student does on our test frames. Green boxes are lights it found at the cutoff where 95% of its detections are right, red boxes are ones it missed, with its score. Most of the misses are not hard, they're obvious lights scoring right under the bar.

Contact sheet of traffic light crops, top rows with green boxes, lower rows with red boxes and scores.
Student v4 on the hand-labeled comma test. Green boxes: found. Red boxes: missed, with the score it gave.

And here's all of it on one chart. Real comma frames, the lamp bloom, and Kellen's yellows took the small model from finding 40% of the lights at 95% precision to 62%. Across every model we trained, none of them ever read a red light as green on this set, and v4 got the color right on every light it found.

Bar chart: Teacher 1 0.30 and 0.14, Student v2 0.40 and 0.23, Teacher 3 0.71 and 0.29, Student v3 0.52 and 0.31, Student v4 0.62 and 0.38.
How many lights each model finds on the comma test while keeping 95% (blue) or 99% (orange) of its detections correct.

Night

Then we looked at night, and it got scary. These are the student's most confident night "false positives" on LISA's test set. Some are real lights again, but a bunch of them are the backs of trucks, two red tail lights stacked on top of each other, which looks a lot like a red signal. If that held up, it meant phantom stops behind a truck at night.

Contact sheet of night crops, including several pairs of stacked red truck tail lights with scores around 0.6.
The student's most confident night "false positives" on LISA's test set. Some are unlabeled real lights, some are the backs of trucks.

So we checked. We took 205 of its confident night detections that LISA had no label for, put them on our labeling site, and Kellen and I sorted every one of them by hand (I did 95 of the 142 frames, Kellen did the other 47). Only 48 turned out to not be traffic lights, mostly business signs, neon, glare and a few tail lights. Over 100 were real lights that LISA never labeled. Again.

Two contact sheets of night crops, the top labeled not a light, the bottom labeled real green or real red.
Our night check. Top: the 48 that weren't traffic lights. Bottom: the most confident of the real lights LISA never labeled.

We did mess up the site though. It didn't show LISA's existing boxes, so we re-drew about 284 lights that were already labeled. Every one we drew agreed with LISA's color so no harm done, but next time the page shows what's already there.

We fed all of it back in, 160 new lights and 48 junk boxes, and trained v5. v5 was worse. On the comma test it dropped from 62% to 44%, and at night it barely moved, so v4 stays.

After that I watched it drive (more on that below) and made the call that for now we're only doing clear daytime driving. It's a supervised experimental system with a person behind the wheel, and I'd rather have a daytime model I trust than a night model I don't, so night and rain wait.

Why it's never sure

Every chart had the same weird thing in it. Real lights usually scored somewhere between 0.5 and 0.7, with a median around 0.58, and junk sat around 0.4, which doesn't leave a lot of room.

Turns out the model splits every light between two neighboring cells in its output grid. The training target marks one cell as the light and treats the cells right next to it as background, and for thin lights the model can't really tell which of the two is the "right" one, so it hedges. That's the "cell 0.63 + nbr 0.59" in the close-ups below.

That wrecks the "safe" setting. We calibrated a cutoff for 99% precision, 0.71, and on a real continuous drive it catches 3% of the lights. So the "safe" setting is basically the same as it being off.

We tried a second-stage checker to fix it, a small network that looks at a crop around each detection and says real light or not. On the test set it helped a lot. On the San Diego drive it rejected a green light in a yellow housing and a red arrow on the side pole, which were both completely real, so that's parked until it stops doing that.

Annotated dashcam frame of a downtown intersection with green lights and red left arrows boxed, plus a heatmap and close-up crops.
v4 on comma's San Diego demo drive. Thick boxes it would act on, thin ones it sees but scores under 0.54. The close-ups show each score split into a cell and its neighbor, and the second-stage checker ("verifier") wrongly rejecting the green in a yellow housing and the red arrow on the left pole.

An actual drive

Test sets are nice but I wanted to see it drive. comma has a public demo route through downtown San Diego, so we ran v4 on minutes 9 through 11 of it, frame by frame, and I watched the whole thing.

24 seconds of v4 on comma's public demo drive through downtown San Diego.

Honestly it does fine. Close lights clear the bar with the right colors, including red left arrows sitting right next to green lights. Farther lights show up but score under the bar until you get closer. Thick boxes are lights it would act on, thin ones it sees but wouldn't.

Annotated dashcam frame of an intersection with four red lights boxed, three over the bar.
Same drive, coming up on a red. The close reds clear the bar, the far ones don't yet.

Watching it isn't really data though, so we also ran it on Bosch's test drive, 9.3 minutes of continuous, fully labeled daytime driving, and scored it the way the car would actually use it. junctiond (the part of TurnPilot that handles intersections) already needs a light to show up 2 frames in a row before it believes it, so we used the same rule.

At a cutoff of 0.54 with that 2-frame hold:

Lowering the cutoff looks tempting since it catches more, but at 0.50 the same drive gets 4 phantom reds in 9 minutes, and at 0.45 it gets 24. We also tried averaging each light's score over 4, 8 or 15 frames instead, and none of them beat just asking for 2 in a row.

So that's what we're shipping for testing: v4, cutoff 0.54, 2-frame hold. The ONNX file is 14.6 MB, and the weights, model card and eval numbers are on Hugging Face.

What's next

A few things are still missing before this goes anywhere near a real Seattle intersection. The model has two outputs that never got any training data: which way a light is facing, and where the stop line is. None of our public datasets label either one, so they're going to come from the same place lightd started, BeamNG, where the game knows exactly where every line is.

Eight BeamNG road views at intersections with green and cyan lines drawn across the lane at the stop line.
Stop lines in BeamNG, where the game knows exactly where each one is.

And eventually the scorecard has to be our own drives, on the actual comma, in Seattle. comma10k got us from 40% to 62%. Past that we need our own footage, and I have no idea yet how many drives that's going to take.

Image credits: crops from the LISA Traffic Light Dataset (CC BY-NC-SA 4.0) and comma10k (MIT), and frames from comma's public demo drive. lightd v4 is released under CC BY-NC-SA 4.0, since LISA and Bosch are non-commercial.

Subscribed. The next post will come to your inbox.