---
title: Forget the Flying Roadster: Tesla's Real Weapon — Taking Apart How a Self-Driving Car Sees and Decides, with a Simulator
url: https://oosioo.com/en/p/%EB%82%A0%EC%95%84%EC%98%A4%EB%A5%B4%EB%8A%94-%EB%A1%9C%EB%93%9C%EC%8A%A4%ED%84%B0-%EB%A7%90%EA%B3%A0-%ED%85%8C%EC%8A%AC%EB%9D%BC%EC%9D%98-%EC%A7%84%EC%A7%9C-%EB%AC%B4%EA%B8%B0
date: 2026-10-18T13:32:34+00:00
author: SYSOP
summary: Tesla's flying Roadster grabbed the headlines, but what made Tesla famous is physical AI such as FSD. Today, open data and open-source simulators let individuals experiment with self-driving too. I sorted what is out there by the hardware it needs, then built a very simple self-driving simulator that sees with a camera and radar, decides by rules and moves the steering wheel and pedals, so perception, decision and control all move together on one screen.
---
# Forget the Flying Roadster: Tesla's Real Weapon — Taking Apart How a Self-Driving Car Sees and Decides, with a Simulator

![The Self-Driving Decision Dissector. On the left, a top-down view of a two-lane road with a white car; the green fan is what the camera sees and the narrow band in the middle is the radar. On the right, the front camera view, with a red box labelled "Truck 0.76 · 60m" on a truck 60 m ahead and the centre of the left lane it is about to move into drawn as a cyan dashed line. Below, the decision panel reads "Change to left lane"](/uploads/3cf4cdd229451c2a.webp)

On 15 October, Tesla finally unveiled the second-generation Roadster, nine years late. What people talked about afterwards was not the spec sheet. The car drove up an upside-down ramp, spun in mid-air, and used compressed-air thrusters to hop over a Cybercab and a Semi before landing. Elon Musk said safety was not the main goal, and YouTuber Marques Brownlee asked whether it was a prank. People came to see a car and left having watched a stunt.

That kind of stunt is not what made Tesla Tesla. What did is FSD, the system that reads the road through eight cameras and drives the car by itself. These days it goes by the name **physical AI**: AI that moves wheels and a steering wheel rather than words on a screen.

This series collects things built by giving AI a line or two of instructions. People say training an AI now takes just a few clicks. If so, can an individual get their hands on a problem as big as self-driving, using open data and open-source tools? That is what this episode checks.

## What I set out to do

1. Find the self-driving simulators, open models and datasets an individual can download and run, including how much hardware they need and what their licences allow.
2. Build a **very simple** self-driving car that sees with a camera and other sensors, makes decisions and moves. Inside it, show what the sensors saw (perception), what the rules decided (decision) and how the steering wheel and pedals moved (control), **all on one screen at the same time**.
3. Make it run right here in this article, in the browser, with nothing to install.

I left the research to AI agents and made them attach a primary source to every number. Where the only source is press coverage, the text says so.

## Where Tesla stands now

Briefly. Tesla's numbers mix company claims with press reports, so I have only kept what is confirmed.

- **FSD is a single neural network from camera to steering wheel.** From v12 in 2024, Tesla says it ripped out hand-written rule code and replaced it with a network that takes camera video in and puts steering and acceleration out. The latest release, pushed out in early October, is in the v14.3 line ([release notes](https://www.notateslaapp.com/software-updates/version/2026.38.5/release-notes)).
- **Driverless robotaxis are still small.** Tesla [disclosed](https://www.sec.gov/Archives/edgar/data/1318605/000162828026003837/exhibit991.htm) that it removed safety monitors from some cars in Austin in January 2026, but as of March only a handful of vehicles were running unsupervised ([Electrek](https://electrek.co/2026/03/31/tesla-expands-unsupervised-robotaxi-service-area-still-only-handful-vehicles/)).
- **Supervised FSD runs in Korea too.** It arrived in November 2025, starting with US-built Model S and X. The driver has to keep watching at all times.

For comparison, as of September 2026 Waymo carries about 500,000 paid rides a week with roughly 4,000 vehicles ([TechCrunch](https://techcrunch.com/2026/09/24/waymo-is-scaling-fast-heres-what-the-fleet-data-shows/)). Waymo uses lidar and high-definition maps; Tesla uses cameras only. Different routes, but both end up answering the same question: **what can I see right now, and so what do I do?**

## What an individual can get their hands on

There turned out to be more than I expected. Here they are, sorted by the hardware you need.

![Bar chart of GitHub stars for self-driving open-source projects an individual can download and run. openpilot 63,924, CARLA 14,505, Autoware 12,165, highway-env 3,330, Alpamayo 2,039, Bench2Drive 1,964, MetaDrive 1,261, NAVSIM 1,131. Projects that run in real cars are red, simulators blue, open driving models amber, benchmarks green](/uploads/d27465bc517d80ac.webp)

### If you only have a laptop

- **[highway-env](https://github.com/Farama-Foundation/HighwayEnv)**: A top-down 2D environment for motorways, junctions and parking. Widely used as a first step into reinforcement learning. Runs without a GPU. MIT licence; version 1.12.1 came out in August.
- **[MetaDrive](https://github.com/metadriverse/metadrive)**: A lightweight 3D simulator that generates roads procedurally. It can also load and replay real driving logs. Apache 2.0.

### If you have a gaming graphics card

- **[CARLA](https://github.com/carla-simulator/carla)**: The standard urban simulator for self-driving research. You fit cameras, lidar and radar and drive around a city. Since version 0.10 moved to Unreal Engine 5 at the end of 2024, it recommends 16 GB of VRAM or more. On top of it sits [Bench2Drive](https://arxiv.org/html/2406.03877v3), which scores driving on 220 short scenarios.
- **[NVIDIA Alpamayo](https://nvidianews.nvidia.com/news/alpamayo-autonomous-vehicle-development)**: A 10-billion-parameter driving model unveiled at CES in January. Feed it camera video and it returns a trajectory for the next 6.4 seconds along with **a written explanation of why it is going that way**. NVIDIA also released a simulator, AlpaSim, and 1,700 hours of driving data. Running it takes a 24 GB card (RTX 3090 or 4090 class). The weights of the first version were released for non-commercial use only.
- **[NAVSIM](https://arxiv.org/pdf/2510.04333)**: Scores a model's plans against real driving logs. Its strength is that it uses real sensor recordings rather than simulator graphics.

### If you have a real car or a model car

- **[openpilot](https://github.com/commaai/openpilot)**: Open-source driver assistance from comma.ai. Fit their device to a car and it handles lane keeping and speed. It supports [more than 350 models](https://comma.ai/vehicles), with plenty of Hyundai, Kia and Genesis cars among them. Version 0.11.2, released on 14 October, brought in an 880-million-parameter driving model. The driving model in 0.11.0, released in March, was trained **entirely inside a learned simulator rather than on real roads** ([release notes](https://github.com/commaai/openpilot/blob/master/RELEASES.md)). This goes into a real car, so check your country's laws and your insurance before using it on public roads.
- **[Donkeycar](https://docs.donkeycar.com/)**: A self-driving model car built from a Raspberry Pi and an RC car. You drive it round a few laps to collect camera footage and steering, train on that, and it drives itself. It is the best way to experience one-click training first-hand.
- **RoboRacer (formerly F1TENTH)**: 1/10-scale race cars with lidar. More than 80 university labs enter its competitions at robotics conferences each year.
- **AWS DeepRacer**, once a popular way in, [shut down](https://aws.amazon.com/deepracer/faqs/) its console service and device sales in December 2025. It now lives on only as open source you deploy into your own account.

### Data

| Dataset | What is in it | What you may do with it |
|---|---|---|
| [comma2k19](https://github.com/commaai/comma2k19) | 33 hours of California motorway driving; camera, GPS, CAN | Repository is MIT; no licence stated for the data itself |
| [KITTI](https://www.cvlibs.net/datasets/kitti/) | Karlsruhe, Germany, 2012; camera and lidar | Non-commercial (CC BY-NC-SA 3.0) |
| [nuScenes](https://www.nuscenes.org/terms-of-use) | 1,000 scenes in Boston and Singapore; 360° sensors | Non-commercial, after registering |
| [Waymo Open Dataset](https://waymo.com/open/terms/) | Several US cities; camera and lidar; long-tail scenarios | Non-commercial; models trained on it may not be used to operate vehicles |
| [Argoverse 2](https://www.argoverse.org/about.html) | Six US cities; built for trajectory prediction | Non-commercial (CC BY-NC-SA 4.0) |
| [NVIDIA PhysicalAI-AV](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles) | 1,700 hours in 25 countries; seven cameras and lidar; 133 TB | Custom terms; no redistribution |
| [AI Hub self-driving](https://aihub.or.kr/aihubdata/data/view.do?dataSetSn=71590) | Korean expressway video and lidar, plus bad weather, parking and more | Korean residents only, after approval |

There is plenty for study and experiments. But almost all of it is **non-commercial**. If you want to sell a model built on this data or put it in a real car, read the terms again first. If you need Korean roads, AI Hub is close to the only way in.

## Two approaches: split it up, or learn it whole

There are broadly two ways to build a self-driving car.

**Split it up (modular).** Perception, decision and control are built separately. You find lane lines and cars in the camera image, decide what to do with rules or a planning algorithm, and a controller turns that into a steering angle and pedal positions. Every stage leaves a record of what it saw and why it acted. The catch is that the more rules you add, the more they collide.

**Learn it whole (end-to-end).** One neural network, trained on human driving logs, takes video in and puts steering and acceleration straight out. Tesla FSD since v12 and openpilot take this route. If a situation is in the data, the network learns it even when nobody managed to write a rule for it. The catch is that **why it did something** is buried in a mass of numbers. That gap is exactly why Alpamayo was built to output its reasons in words alongside its trajectory.

What I built deliberately takes the first route. To see what the second route is replacing, you first need to see the first route's decisions with your own eyes.

## What I built: the Self-Driving Decision Dissector

Here is the simulator. It starts driving straight away. Use the red buttons at the bottom to inject situations.

```map /embed/av-pipeline-2026-en.html 9:10
The Self-Driving Decision Dissector. A white car drives itself round a one-way, two-lane loop 1,028 m long. On the left is a top-down view of the road with camera and radar coverage; on the right, the front camera view and what it detected. The three panels below are perception (values the sensors read), decision (the rules checked from the top, and the current decision) and control (steering angle, throttle and brake). The chart shows the last 20 seconds of speed and of commanded versus actual steering, with the background colour showing the decision at each moment. The buttons "Lead car brakes", "Pedestrian darts out", "Car cuts in" and "Camera blocked" inject situations.
```

The road is a one-way, two-lane loop 1,028 m round, made of straights and curves joined together. There is one set of traffic lights and one pedestrian crossing, and the car shares the road with a truck doing 40 km/h and three other cars. The speed limit is 60 km/h.

**① Perceive: two sensors.** The front camera sees 35° either side, out to 70 m. It reads the lane position and looks for cars, people and traffic lights, growing less certain with distance. The radar is narrow, 9° either side, but reaches 120 m and measures distance and speed precisely. So a distant truck is picked up first by "radar only", and becomes "camera+radar" as it gets closer. I deliberately added noise to the lane position the camera reads. That is why the steering line on the chart jitters slightly.

**② Decide: a rule table.** The decision panel runs through seven rules from the top. The higher a rule, the closer it is to safety.

1. Can the camera be trusted? If confidence falls below 0.6, slow to 30 km/h and hand over to the driver.
2. Is time to collision (TTC) under 1.6 seconds? If so, override every other decision and brake as hard as possible.
3. Is there a person ahead, or one likely to step in?
4. Is the light red, or amber with enough room to stop comfortably?
5. Is there a car ahead in the same lane within 80 m? If so, follow it at a set gap.
6. Is the car ahead more than 8 km/h slower, the next lane clear ahead and behind, and no light, crossing or bend coming up? If so, overtake on the left, and return when the outer lane is clear.
7. Is there a bend within 50 m? Slow in advance to a speed that keeps sideways acceleration under 2 m/s².

Each rule asks for an acceleration, and the **most cautious** one wins. The large label in the decision panel is the rule currently winning.

**③ Control: steering and pedals.** Steering uses pure pursuit. It aims at a point on the centre of the target lane, a distance ahead proportional to speed (6 to 25 m), and works out the wheel angle that traces an arc through that point. The method comes from a 1992 Carnegie Mellon report and still appears unchanged in introductory textbooks. Acceleration when following a car is set by IDM, a model from traffic research. Finally, **actuators cannot follow commands instantly.** The steering lags by 0.12 seconds and the pedals by 0.2 seconds, and the steering also has a rate limit. In the steering wheel graphic in the control panel, the cyan mark is the command and the real wheel chases it.

## Running it

![An 80-second driving log. From the top: speed (target speed dashed, actual speed solid), steering wheel angle (command and actual), and throttle and brake. The background shows the decision at each moment. At 7.6 s the car spots the truck and sets its gap; at 12.2 s it overtakes in the left lane and slows to 40 km/h for the bend. At 43.6 s it slows to 40 km/h for a person on the crossing, then moves off once they have crossed, and at 51.6 s it returns to the outer lane. At 71 s a person darts out, the brake goes in close to 90%, and speed drops from 60 to 7 km/h](/uploads/7abc49423ad1985e.webp)

I recorded the first 80 seconds exactly as they ran. At 71 seconds I pressed "Pedestrian darts out" once.

- **7.6 s**: The radar picks up the truck 80 m ahead first. The decision switches to "Follow lead car" and speed drops to 54 km/h.
- **12.2 s**: The truck is slow at 40 km/h and the left lane is clear. "Change to left lane". The steering wheel turns about 10° and comes back.
- **From 15 s**: A bend with a 60 m radius. The target speed drops first (dashed), and the actual speed follows. Through the bend the steering wheel holds at around 40°.
- **43.6 s**: Someone is on the crossing. The car starts slowing 60 m away, gets down to 40 km/h, and speeds up again once the person has crossed.
- **71 s**: A person darts out from the kerb about 30 m ahead. Within 0.2 seconds "Under 1.6 s to impact" fires and the brake goes in close to 90%. While the car drops from 60 to 7 km/h, the person gets clear of the lane.

What stands out most on the chart is **the gap between the moment a decision changes and the moment the car moves**. The target speed bends like a staircase, but the actual speed takes several seconds to follow. However good the decision, the car only moves as fast as physics and its actuators allow.

## Where it went wrong

I took the rules in my first version and pressed the buttons one by one. Here is what I fixed.

- **It pulled in right in front of the slow truck.** The return rule only checked whether the 40 m ahead in the outer lane was clear. My speed had dropped in the bend, so I had not got past the truck, yet with the truck 43 m ahead the rule called the lane clear and pulled in behind it. Seventeen seconds later it overtook the same truck again. I changed it so that if the car ahead is slower than me, I only return once it is more than 100 m away.
- **It hit a car cutting in.** A car from the next lane moved in 9 m ahead, but it only counted as "the car ahead" once it was inside my lane. By then it was too late. I changed it to use the car's sideways speed and treat any car that will be in my lane within 1.5 seconds as the car ahead. Now the log reads "From next lane, merging car moving in → slow early".
- **It hit a person darting out.** A person standing at the kerb was outside the lane, so it was ignored. If that person then runs out at 2.4 m/s, it is already too late. I made it predict where a pedestrian will be two seconds ahead in the direction they are moving, and pick up anyone about to come in front of me.
- **It drove past a person on the crossing.** It only looked at people in front of my own lane, so it passed someone who had almost finished crossing, at the far end of the crossing, at 59 km/h. Korean road traffic law requires drivers to stop when a pedestrian is crossing or about to cross at a crossing. I changed it so that anyone on the crossing is a reason to stop, whatever lane they are in.

All four were fixed by **adding one more rule**. But there are only four buttons. On a real road a ball rolls out, cargo falls off a lorry, a police officer waves you on by hand. Patching a rule onto every scene is a road with no end. That is why the industry has moved towards learning from whole human driving logs. It is also why Waymo's open data gathers scenes that occur less than 0.03% of the time into a separate [evaluation set](https://waymo.com/open/challenges/2025/e2e-driving/).

In this simulator, on the other hand, every fix is visible on screen. Why the car stopped, and which rule won, stays in the decision panel. In a network that learned everything end to end, you cannot see that.

## Why one line was enough

Once again, all the ingredients were public. Pure pursuit dates from 1992 and IDM from 2000, and emergency braking triggered by time to collision is how European new-car safety tests are run. How to join a road from straights and arcs, and how to draw a camera's perspective, are in the textbooks too. The AI assembled them into a single file. That is why it runs in one file with no outside libraries.

That is as far as one line got. The four collisions above did not show up until I pressed the buttons. The code ran without problems from the start. But running is not the same as being right.

## If you want to try it

- **Make the decisions visible first.** If the model shows "why" on screen when it stops, you find the mistakes quickly. All four fixes here started from a single line in the log.
- **Throw accidents at it on purpose.** Left alone, it drives round happily with nothing going wrong. The holes in the rules only appeared after I injected hard braking, cut-ins and people darting out.
- **Plot the command and the result separately.** Put the steering command next to the actual steering, and the target speed next to the actual speed, and you can tell whether the decision was late or the car was.
- **The next step is highway-env or MetaDrive.** Swap this simulator's decision panel for a single neural network and run reinforcement learning, and you can see for yourself where rules and learning part ways. A laptop will do.

## What it can't do

- **It does not decide from camera images.** The camera view is a drawing for display. The actual perception fakes it by adding noise to the true positions the simulator already knows. Finding lane lines and cars in real video is the hardest part of self-driving, and that part is missing.
- **The road is simple.** One loop of one-way, two-lane road. No junctions, no turning across traffic, no oncoming cars, no parking.
- **The car is a bicycle model.** No tyre slip, no road surface, no weight transfer. Press the brake and it simply stops.
- **The rule numbers are chosen values.** Thresholds like TTC 1.6 seconds or the 8 km/h overtaking condition are plausible values I picked. They are not taken from a real car.
- **Do not use it in a real car.** It is a picture for learning.

The flying Roadster will soon be forgotten. What lasts is how trustworthy you can make these three panels: seeing, deciding and moving. Press one of the buttons above. You will see exactly why the car stops in the decision panel.

### If you enjoyed this, please leave a comment.

## Sources

- Roadster reveal: [TechCrunch](https://techcrunch.com/2026/10/15/tesla-reveals-spacex-powered-roadster-that-flies-and-falls-with-style/) (15 October 2026), [Electrek](https://electrek.co/2026/10/13/tesla-roadster-event-thursday-nine-years-delays/) (13 October 2026)
- openpilot release notes: [RELEASES.md](https://github.com/commaai/openpilot/blob/master/RELEASES.md) (0.11.0 on 17 March 2026, 0.11.2 on 14 October 2026)
- NVIDIA Alpamayo: [announcement](https://nvidianews.nvidia.com/news/alpamayo-autonomous-vehicle-development) (5 January 2026), [model card](https://huggingface.co/nvidia/Alpamayo-R1-10B)
- CARLA 0.10: [announcement](https://carla.org/2024/12/19/release-0.10.0), [recommended specs](https://carla-ue5.readthedocs.io/en/latest/start_quickstart/)
- Pure pursuit: R. C. Coulter, Implementation of the Pure Pursuit Path Tracking Algorithm, [CMU-RI-TR-92-01](https://www.ri.cmu.edu/publications/implementation-of-the-pure-pursuit-path-tracking-algorithm/) (1992)
- IDM: Treiber, Hennecke and Helbing, Congested traffic states in empirical observations and microscopic simulations, [Phys. Rev. E 62](https://arxiv.org/abs/cond-mat/0002177) (2000)
- GitHub star counts were taken from the GitHub API on 18 October 2026
- The simulator images and driving log were all produced with this article's simulator