Waymo’s self-driving cars have driven 10 million miles on public roads, the company announced on Wednesday. That puts the company way, way ahead of any of its rivals. Only one other company has announced more than a million miles of testing. And that was Uber, which announced 2 million miles last year but was forced to freeze its public testing after a deadly crash in March.
The milestone is one of many signs that Waymo has a sizable lead in the race to commercialize driverless car technology. Waymo says it is planning to launch a taxi service in the Phoenix area before the end of the year. One of Waymo’s leading rivals, GM’s Cruise, is aiming to launch a commercial taxi service next year. Most other companies working on driverless car technology aren’t planning commercial launches until the early 2020s.
Rivals say testing miles are an overrated metric
If you talk to Waymo’s rivals, most of them are quick to dismiss the significance of testing miles. They argue that quality is far more important than quantity when it comes to driverless car testing. It’s easy to rack up a lot of miles driving the same routes over and over again, they say, but that may not actually prove very much about the quality of its technology. Uber’s deadly car crash is a case in point: at the time of the crash, Uber was second only to Waymo in testing miles, but its technology was deeply flawed.
It’s an argument that has been made forcefully by Kyle Vogt, CEO of Cruise, the GM-backed driverless car company that is arguably Waymo’s most formidable competitor. Cruise’s latest report to California regulators, filed early this year, showed that the company had logged only 131,000 miles on public roads between December 2016 and November 2017 (for comparison, Waymo accumulated 350,000 in California and many more in Arizona and elsewhere). But the bulk of that testing was in urban San Francisco, and Vogt pointed out in a blog post last year that San Francisco is a much more challenging environment for driverless cars than the Phoenix area, where Waymo does a lot of its testing.
“Our vehicles encounter challenging (and often absurd) situations up to more often than other places self-driving cars are tested,” Vogt wrote. Vogt Specifically compared Cruise’s San Francisco testing to the Phoenix suburbs, where Waymo (and at the time, Uber) were performing a lot of their tests.
There’s a lot of merit to this argument. A major goal of real-world testing, especially early in the development process, is to encounter difficult and unusual situations. And a car is obviously going to encounter more tricky situations per mile in San Francisco than in Chandler, Arizona.
Racking up miles is essential before a commercial launch
But the goal of testing changes a bit when a company is getting close to releasing a commercial service. In late-stage testing, one goal is still to encounter difficult and unusual driving situations that can be put into a simulator to verify that all future software versions can handle them. But late-stage testing also serves as a “dress rehearsal” for all aspects of the soon-to-launch commercial system.
Late-stage testing helps ensure that sensors and other safety-critical vehicle hardware have a low failure rate. It also helps ensure that remote fleet operators are well-trained and the organization is ready to handle any contingency. It allows the company to verify that cars will use safe locations to pick up and drop off passengers. And it allows the company to identify other non-safety issues that might crop up, like whether there are enough vehicles available for rush-hour service and whether there are adequate plans in place for vehicle cleaning and maintenance.
This kind of late-stage testing should be as similar as possible to the final, shipping product. It’s also important to do it while carrying actual passengers on “real” routes. That will inevitably mean repeatedly driving the same routes, since actual customers tend to go to the same places (work, school, the local grocery store) over and over again. This kind of testing won’t provide as much useful data from a pure software engineering perspective, since many of the miles will be on routes and situations the software has already mastered.
But it’s also hard to think of an alternative. A driverless taxi service is a complex, integrated system, with hardware, software, and Waymo staffers all working together to get passengers from point A to point B. Any testing regime that doesn’t closely mirror the conditions of the final product creates the danger that the testers will fail to uncover problems that arise from the interactions of these systems—for example, Waymo employees failing to replace defective components or making errors due to confusing back-end software interfaces.
Ultimately, there’s just no substitute for actually putting the system through its paces and seeing what kind of problems crop up.
People sometimes criticize Waymo for focusing so heavily on Phoenix-area testing, arguing that this will leave Waymo’s cars unprepared for the more challenging driving environments elsewhere in the country. But that’s precisely the point: Waymo’s commercial taxi service will initially only be available in the Phoenix area.
It doesn’t matter how the system would perform in Detroit snow or Manhattan traffic jams, because, at least initially, Waymo won’t be offering commercial service anywhere other than the Phoenix metropolitan area. The company will presumably do a lot more testing in those other places before launching services there.
In short, what’s striking about Waymo isn’t just that it has done far more miles of testing than any of its rivals, it’s that the nature of its testing is different. Most of Waymo’s competitors are still doing early-stage tests where the goal is to accumulate a library of unusual driving scenarios to plug into their simulators and improve the software. Their testing is focused on providing engineers with the data they need to make the software better.
Waymo, by contrast, believes that its software is very close to being ready for commercial launch, and that has led the company to focus on more comprehensive testing—testing designed to validate Waymo’s hardware and organization as well as its software. The fact that no one else is really doing that seems like a sign of how far ahead Waymo is.