Spencer Platt / Getty Images
The National Highway Traffic Safety Authority has an "egg on the face" after a small research and consulting firm called Quality Control Systems has produced a devastating critique of an agency report published in 2017 stating that Tesla's autopilot reverses the number of crashes 40 percent decreased. The new analysis appears almost two years after the original report because QCS had to sue NHTSA under the Freedom of Information Act to obtain the data underlying the agency's findings. In his report, QCS highlights errors in the NHTSA methodology that are serious enough to completely discredit the 40 percent figure that Tesla has cited on several occasions over the last two years.
NHTSA conducted its autopilot safety study following the fatal crash of Tesla owner Josh Brown in 2016. Autopilot – more specifically: Tesla's tracking function called Autosteer – was active at the time of the crash, and Brown ignored multiple warnings his hands back on the wheel. Critics asked if the autopilot actually made the owners of Tesla less secure by encouraging them to pay less attention to the road.
NHTSA's finding in 2017 that Autosteer lowered crash rates by 40 percent seemed to ease that concern. When another Tesla customer, Walter Huang, died in an auto-storm crash last March, Tesla quoted NHTSA's 40 percent figure in a blog post defending the technology. A few weeks later, Elen Musk, CEO of Tesla, scolded the reporters for focusing on crashes and not pointing out the safety benefits of autopilot.
"You should write a story about how autonomous cars are really safe," Musk said in a May 2018 appeal. "But that's not a story people want to click on, they write riotous headlines that mislead readers in general. "
Now the full NHTSA dataset is available, and if anything, that seems to contradict Musk's claims. The majority of vehicles in the Tesla dataset suffered from missing data or other issues that made it impossible to say whether Autosteer activation increased or decreased the crash rate. When the QCS focused on 5,714 vehicles whose data did not suffer from these issues, it found that the activation of the Autosteer actually increased the accident rates by 59 percent.
In a sense, these are old news. The NHTSA distanced itself from its own findings last May, describing it as a "fleeting comparison" that "does not rate the effectiveness of Autosteer technology." In addition, the NHTSA report focused on version 1 of the autopilot hardware that Tesla has not sold since 2016.
However, these new findings are relevant to a broader debate on how the federal government monitors driver assistance systems such as autopilot. By publishing this 40 percent figure, NHTSA granted unjustified authorization to Tesla's autopilot technology. NHTSA fought to prevent the release of data that could help the public to independently evaluate these results so that Tesla can quote the numbers for another year. In other words, NHTSA seemed to focus more on protecting Tesla from embarrassment than protecting the public from potentially unsafe vehicle technologies.
Serious mathematical errors made the autopilot look better
Tesla started shipping vehicles with autopilot hardware in 2014. However, the lane keeping system Autosteer was activated only in October 2015. This was a natural experiment: by comparing the accident rates of the same vehicle before and after October 2015, NHTSA was able to estimate how the technology affected safety.
Therefore, NHTSA Tesla asked for data on cars sold between 2014 and 2016 – how many kilometers they covered before and after the autopilot upgrade, and whether a crash occurred before or after the autopilot was activated (as measured by the airbag deployment). Tesla has unusually complete data on such things as its vehicles are wirelessly connected to Tesla's headquarters.
The calculations of the results of NHTSA should be straightforward. The Agency summarized its findings in the January 2017 report:
NHTS To calculate a crash rate, take the number of crashes and divide them by the number of miles driven. NHTSA ran this calculation twice – once for miles before the Autosteer upgrade and again for miles. NHTSA found that pre-Autosteer crashes occurred more frequently and the rate dropped 40 percent after activation of the technology.
In such a calculation, it is important that the numerator and denominator are drawn from the same group of data points. If the miles of a particular car are not in the denominator, crashes for the same car can not be in the meter – otherwise the results are meaningless.
NHTSA has done just that, according to QCS. Tesla transmitted NHTSA data to 43,781 vehicles, but 29,051 of these vehicles lacked the required data fields to calculate how many kilometers these vehicles traveled before activating the Autosteer. NHTSA handled this by counting these cars as pre-Autosteer driven miles. NHTSA counted these vehicles but with 18 crashes in front of the Autosteer – more than 20 percent of the total of 86 recorded before the Autosteer crashes in the record. The result was a clear overestimation of the crash rate before the Tesla Autosteer.
Other vehicles in the record had a different, more subtle problem. Tesla provided two different data points for vehicles: the last mileage before the installation of the Autosteer and the first mileage after. Ideally, these readings would be identical. However, with 8,881 vehicles, there was a significant difference between these numbers. These cars have covered millions of miles in which it is unknown whether Autosteer was active or not.
NHTSA has decided not to include these miles in the denominator for calculation before or after Autosteer. The result is the inflation of the crash rate for the "before" and "after" periods. This almost certainly made the results less accurate, but it's impossible to know if the autopilot looks better or worse on the web.
It is only possible to calculate accurate crash rates for vehicles that have complete data and have no gap between pre-Autosteer and post-Autosteer rangefinders. Tesla's dataset included only 5,714 vehicles. When QCS Director Randy Whitfield calculated the numbers for these vehicles, he found that the crash rate per mile increased by 59 percent after Tesla had activated the Autosteer technology.
Does that mean that Autosteer makes crashes 59 percent more likely? Probably not. These 5,714 vehicles represent only a small part of the Tesla fleet, and there is no way to know if they are representative. And that's the point: it is ruthless to draw conclusions from such erroneous data. NHTSA would either have asked Tesla for further information or should have removed all of this from the report.
NHTSA withheld its data from the public at the request of Tesla
The misinformation in the NHTSA report could have been corrected much faster if NHTSA had decided to make its data and methods transparent. The QCS made a request in February 2017, about a month after the report was published, for the data and methodology underlying the NHTSA conclusions. If NHTSA provided the information in a timely manner, the problems with NHTSA's calculations would probably have been detected quickly. Tesla could not have cited her more than a year after its release.
Instead, NHTSA fought QCS 'FOIA request after Tesla said that the data was confidential and would cause Tesla a competitive claim when it was released. QCS sued the agency in July 2017. In September 2018, a federal judge rejected most of NHTSA's arguments and cleared the way for NHTSA to release the information to QCS late last year.
QCS says it has no financial stake in the autopilot controversy and has not received external support for its litigation. "It seemed important," Whitfield said. "We had experience with FOIAing data from NHTSA and knew a good lawyer."
We asked NHTSA on Monday morning for comments on the QCS analysis. At their request, we have delayed the publication of this story by 24 hours to give them time to prepare an answer. The organization finally sent us, "The agency is watching the report published by Quality Control Systems Corp. with interest and may comment accordingly."
We also asked Tesla for a comment. "The QCS analysis discarded the data from all but 5,714 vehicles out of a total of 43,781 vehicles in the dataset we provided to NHTSA in 2016," a company spokeswoman wrote. "Given the dramatic increase in the number of Tesla vehicles on the road, their analysis today accounts for approximately 0.5 percent of Tesla vehicles driven so far, and about 1 percent of Tesla 's total mileage enabled Autopilot.
Tesla also pointed out that the 2017 NHTSA report had other positive things to say than 40% beyond the autopilot. NHTSA wrote that it had "found no defects in the design or performance of the AEB or autopilot systems," nor "identified incidents where the systems did not function as intended."
Tesla also announced a new quarterly safety report, which Tesla now publishes on its website. This report shows that Tesla autopilot-enabled vehicles incur fewer accidents per mile than Tesla autopilot-enabled vehicles. These in turn suffer less accidents per mile than an average car on the road.
But Whitfield also rejects this data.
"The statistics are not controlled for many of the implications we know are important," Whitfield told Ars in a telephone interview on Monday. The autopilot deployment should be limited to highways that experience fewer accidents per mile than other roads. The fact that there are fewer crashes per mile when the autopilot is activated does not necessarily prove that the autopilot makes the rides safer. It could simply reflect the fact that crashes on highways are rarely encountered per kilometer.
When comparing Tesla and non-Tesla vehicles, new and high-end vehicles typically have a much lower accident rate than cars. Tesla's lower accident rates may reflect the fact that most Tesla vehicles are newer than the average. This means that they will have fewer problems that inevitably arise with the age of the cars.
Due to the high cost of Tesla's vehicles, Tesla drivers are probably richer and older than the average driver. Middle-aged drivers tend to be safer than young drivers, and more affluent drivers can usually afford regular maintenance to avoid safety hazards such as bare tires. Tesla's relatively low accident rate may reflect the demographic evolution of the customer base more than the safety features of its vehicles.
As far as we know, Tesla has not provided independent experts with up-to-date accident data that could potentially influence these types of factors in order to rigorously assess the safety of the autopilot.