Tesla vs. NYT: Do the Data Really Tell All?

Tweet about this on TwitterShare on Facebook26Share on Google+5Share on LinkedIn1Email this to someone

I've enjoyed so far the back and forth between Tesla Motors and New York Times reporter John Broder. The short version is

  • Broder tested one of Tesla's new Model S all-electric sedans on a drive from Washington, D.C. to Groton, CT. Part of the reason for this specific trip was to make use of Tesla's new supercharger stations along the route (one in Delaware and one in Connecticut).
  • Broder's trip appeared to have some bumps, including running out of electricity at one point and requiring a tow.
  • After the review was published in the New York Times, Elon Musk, the CEO/Founder of Tesla, was apparently livid. He published a detailed response on the Tesla blog explaining that what Broder wrote in his review was not true and that "he simply did not accurately capture what happened and worked very hard to force our car to stop running".
  • Broder has since responded to Musk's response with further explanation.

Of course, the most interesting aspect of Musk's response on the Tesla blog was that he published the data collected by the car during Broder's test drive. When revelations of this data came about, I thought it was a bit creepy, but Musk makes clear in his post that they require data collection for all reviewers because of a previous bad experience. So, the fact that data were being collected on speed, cabin temperature, battery charge %, and rated range remaining, was presumably known to all, especially Broder. Given that you know Big Brother Musk is watching, it seems odd to deliberately lie in a widely read publication like the Times.

Having read the original article, Musk's response, and Broder's rebuttal, one things is clear to me--there's more than one way to see the data. The challenge here is that Broder had the car, but not the data, so had to rely on his personal recollection and notes. Musk has the data, but wasn't there, and so has to rely on peering at graphs to interpret what happened on the trip.

One graph in particular was fascinating. Musk shows a periodic-looking segment of the speed graph and concludes

Instead of plugging in the car, he drove in circles for over half a mile in a tiny, 100-space parking lot. When the Model S valiantly refused to die, he eventually plugged it in.

Broder claims

I drove around the Milford service plaza in the dark looking for the Supercharger, which is not prominently marked. I was not trying to drain the battery. (It was already on reserve power.) As soon as I found the Supercharger, I plugged the car in.

Okay, so who's right? Isn't the data supposed to settle this?

In a few other cases in this story, the data support both people. In particular, it seems that there was some serious miscommunication between Broder and Tesla's staff. I'm sure they also have recordings of those telephone calls too but they were not reproduced in Musk's response.

The bottom line here, in my opinion, is that sometimes the data don't tell all, especially "big data". In the end, data are one thing, interpretation is another. Tesla had reams of black-box data from the car and yet some of the data still appear to be open to interpretation. My guess is that the data Tesla collects is not collected specifically to root out liars, and so is maybe not optimized for this purpose. Which leads to another key point about big data--they are often used "off-label", i.e. not for the purpose they were originally designed.

I read this story with interest because I actually think Tesla is a fascinating company that makes cool products (that sadly, I could never afford). This episode will surely not be the end of Tesla or of the New York Times, but it illustrates to me that simply "having the data" doesn't necessarily give you what you want.

  • ezracolbert

    Have you ever tried to find something at night, in an unfamiliar place, when you are tired and stressed ?

    What may be totally obvious to someone who has been there before, in the daylight, might appear hard to find to someone else
    assuming that the numbers are accurate - and i don't understand why people take them at face value, without knowing where they come from, seems like gigo - it shows that this tired guy drove around the plaza once or twice to find the charger

    • http://twitter.com/hspter Hilary Parker

      What a strange and hostile comment. I don't think the point of *this* article was whether or not the guy drove around for five minutes looking for the charger. It was just that data don't always tell the full story -- there could be multiple interpretations for that data pattern.

      • ezracolbert

        given the blog coverage, i'm not sure why you find this strange and hostile
        according to the blogosphere, broder *deliberatly* drove around the plaza, to ruin the car, to fit his (broders) preconcieved need to write a hostile story.

        Given this, i don't see how my comment is strange or hostile, but each to his own.
        I also don't understand this wierd attitude that numbers are in and of themselves, presumed accurate - you do recall the 1st iraq war, scuds, patriots and T Postol, don't you ?
        Or, when hometown boosters in Denver were trying to snag the NBA championship, they did a report which forecast more visitors then hotel rooms in all of denver (this was in a very funny pdf about how such forecasts are common)

        on a stats blog, one would expect that the FIRST thing is, what is the integrity and quality of hte data ?
        and we have no idea, as the primary data, and software to crunch it, are not yet public (or no one has bothered to look)

        • http://twitter.com/hspter Hilary Parker

          Oh OK I get it -- you're battling a strawman, rather than actually responding to the content of this post. Apologies for getting in the way!

          • ezracolbert

            in the future, all commenters must strictly comment only on the main point of a blog post.
            Thank you for providing guidance to the rest of us.

  • Thomas Lumley

    Have you seen the Tesla Road Trip: http://allthingsd.com/20130217/tesla-owners-hit-the-road-to-prove-long-distance-can-be-done/

    7 Tesla owners repeated the trip, live-tweeting.

    • ezracolbert

      what, exactly, does this prove ?
      That someone who has an axe to grind (eg, someone who spent a lot of money on an electric car ) gets a result that differs from a possibly unbiased person ?
      that if you do something twice, you don't always get the same result ?
      That know how someone erred the 1st time allows you to do better the second time ?

      That tesla batterys may be manufactured inconsistenly, or have strong nonlinear temperature/performance curves ?

  • http://www.linkedin.com/in/ashujoshi Ashu Joshi

    Data or Big Data can be used to predict/forecast. I agree to this only partially. As long as there are humans, there is nothing about them that can be gauged by data. I am slightly biased towards NYT's John Broder - because humans do not behave rationally. Not everybody, is logical and reasonable in most things they do. Big Data would be "more" useful if it can be used to pick the right irrational behavior humans exhibit.