There have been a mountain of articles about self-driving cars, but little discussion about how they see the world around them. The ability of computers to understand images is still in its infancy – in 2015 there was a lot of talk about how Google was teaching an AI program how to recognize cats within videos.
But obviously a self-driving car has to do a lot better than just ‘seeing’ around it – it needs to paint a 3D picture of everything around it in order to navigate correctly and to avoid problems. It turns out that the primary tool used by self-driving cars is called “Lidar.” Lidar stands for ‘light detection and ranging’ and fits neatly between sonar and radar.
Lidar works by sending out light beams and measuring how long it takes for reflected signals to return, much the same way that a bat sees the world using sonar. Sonar would be fairly useless in a self-driving car since sound waves get distorted in air and only paint an accurate picture for perhaps a dozen feet from the transmitter. That’s great for a bat catching a moth, but not useful for seeing oncoming traffic.
And the radio waves used in radar won’t really work well for self-driving cars. Radar works great for seeing objects far away, like metallic airplanes. But the radio waves pass through many objects (like people) meaning that radar doesn’t create a total picture of the world around it. And radar has problems creating an accurate picture of anything closer than 100 feet.
And that’s where lidar comes in. A lidar device works much like a big radar dish at an airport. It rotates and sends out light signals (actually infrared light signals) and then collects and analyzes the returning echoes to create a picture of the distances to objects around it. Lidar only became practical with modern computer chips which allow the transmitter to ‘rotate’ hundreds of times a second and which possess enough computing power to make sense of the echoed light waves.
And so a self-driving car doesn’t ‘see’ at all. The cars do not rely on standard cameras that try to make sense of the reflected ambient light around the car. The first prototypes of driverless cars tried to do this and could not process or make sense of images fast enough. Instead self-driving cars send out laser light at a specific frequency and then calculates the distance the light travels in every direction to create a picture of the world.
If you want to understand more about what this looks like, consider this Radiohead music video. Most of the images in the video were created with lidar. Don’t pay too much attention to the opening headshots because those are somewhat distorted for artistic effect. But the later images of seeing streets shows you the detail of a lidar image. Unlike the normal images our eyes see, a lidar image is massively more detailed in that the distance to everything in such a picture is known. Our eyeballs basically see in 2D and we use images from two eyes to simulate 3D. But a lidar image is fully 3D and gets full perspective from one transmitter.
Lidar does have limitations. It can be ‘blinded’ by heavy snows and rains. Lidar could be jammed by somebody transmitting a bright signal using the same light frequencies. And so smart cars don’t rely 100% on lidar but also use traditional cameras and sonar using the ultrasound frequencies to complement the lidar images.
Lidar is finding other uses. It’s being used, for example, in helicopters to search for things on the ground. A lidar system can spot a fleeing criminal or a lost child in the woods far more easily than older technologies or human eyeballs. Lidar can also create amazingly detailed images of anything. Archeologists are using it to create permanent images of dig sites during various stages of excavation before objects are removed. It’s not hard to think that within a few years that many traditional surveying techniques will be obsolete and that lidar will be able to locate and plot everything on a building lot, for example, down to the millimeter.