Skip to content ↓

Please note:

MIT and the admissions office will be closed November 24–25 for Thanksgiving break, and will open on November 28.

MIT staff blogger Chris Peterson SM '13

The Illusion Of Validity by Chris Peterson SM '13

look inside your heart. you know it to be true.

In the CollegeConfidential discussion of my blog post The Difficulty With Data, CC poster mihcal1 made the following compelling comment:

So basically, it’s a perfect setup for the Illusion of Validity

Why is MIT’s admissions process better than random? Say you weeded out the un-qualified (the fewer-than-half of applicants insufficiently prepared to do the work at MIT) and then threw dice to stochastically select among the remaining candidates. Would this produce a lesser class?

The link in mihcal1’s post takes you to an article from New York Times magazine by Daniel Kahneman. Kahneman is a pioneer of behavioral economics and the psychology of decision making. He is one of my favorite social scientists, and his work laid the foundation for much of the social science research I love best.

In his article, Kahneman describes his time working as a psychologist for the Israeli Army. They were tasked, among other things, with putting officer candidates through a series of challenges (an application, as it were) to test their leadership potential. They would watch the candidates as they completed challenges, and then they would predict how well they would succeed at officer candidate school.

According to Kahneman:

…as it turned out, despite our certainty about the potential of individual candidates, our forecasts were largely useless. The evidence was overwhelming. Every few months we had a feedback session in which we could compare our evaluations of future cadets with the judgments of their commanders at the officer-training school. The story was always the same: our ability to predict performance at the school was negligible. Our forecasts were better than blind guesses, but not by much.

I thought that what was happening to us was remarkable. The statistical evidence of our failure should have shaken our confidence in our judgments of particular candidates, but it did not. It should also have caused us to moderate our predictions, but it did not. We knew as a general fact that our predictions were little better than random guesses, but we continued to feel and act as if each particular prediction was valid. I was reminded of visual illusions, which remain compelling even when you know that what you see is false. I was so struck by the analogy that I coined a term for our experience: the illusion of validity.

Why, asked mihcal1, were we, as admissions officers, so sure that we were right in our decisions? What made us think our decisions would be better than random guesses? And how can we know?

This is a very good question to ask, and a very difficult one to answer.

Part of the reason it is so difficult to answer is because of the problems I discussed in the last post, which is basically: well, what makes our decisions “better”? How do we know if one applicant is “better” than the other? What does “better” even mean? We could cherrypick any number of metrics that would make the case in our favor. For example, over the last decade or so, our average applicant SAT score has gone up, and our average rate of admission has gone down. You might intepret this to say that we are admitting smarter students, and that we are doing a good job of recruiting applications too, so hey, we’re all going a pretty good job!

Of course, I think those are terrible metrics by which to measure an applicant or an admissions process. What matters isn’t raw SAT score, or how many people we can convince to apply. What matters is making sure that we bring smart students who feel at home here. Who love the community they are in. Who believe in the things that we do here at MIT and who will go out and change the world to be a better place.

As it turns out those things are much, much harder to measure.

Does this mean that our process is no better than random? That all we are doing is admissions shamanism, voodooing behind closed doors of admissions committee before coming out into the light and announcing the signs we’ve read in the application’s entrails?

I don’t think so, for a few different reasons.

One reason is to remember a fundamental limitation of social science, which is that it is situation dependent, and thus it is most usefully and reliably deployed for falsifying specific hypotheses rather than drawing conclusions across contexts.

For example, Kahneman cites research into decades of data which demonstrate that most stock pickers and fund managers basically do no better than random guessing would predict. This sort of question is right in the social science wheelhouse. Hypothesis: variance in skill explains differences in performance between investment managers. Test: do stock pickers routinely perform better than random chance would predict? Result: mostly, no. Hypothesis false, or at least seriously weakened.

But it’s not clear that an admissions process is anything like picking stocks, so it’s also not clear that the same phenomena can be generalized to the work we do. Trying to carry such a slippery situational insight across different contexts is an intellectually dubious exercise.

Another problem with the Army example I alluded to earlier: what’s to say that the psychologists weren’t “better” at picking officers than their future commanders? What does “better” in this context even mean? Without measuring the judgments of the commanders, how could we know? And how would we measure it?

Clearly Kahneman thinks that some people (Israeli Armi commanders) are better at picking some things (future officers) than other people (inexperienced psychologists). And this assumption actually reveals a pretty interesting premise: that there are some real experts. So let’s approach this from another angle: what conditions, according to Kahneman, might make you think that an expert is actually an expert? That a professional is actually good at their job, and not merely reproducing the random and taking credit for it?

Quoth Kahneman:

True intuitive expertise is learned from prolonged experience with good feedback on mistakes. You are probably an expert in guessing your spouse’s mood from one word on the telephone; chess players find a strong move in a single glance at a complex position; and true legends of instant diagnoses are common among physicians. To know whether you can trust a particular intuitive judgment, there are two questions you should ask: Is the environment in which the judgment is made sufficiently regular to enable predictions from the available evidence? The answer is yes for diagnosticians, no for stock pickers. Do the professionals have an adequate opportunity to learn the cues and the regularities? The answer here depends on the professionals’ experience and on the quality and speed with which they discover their mistakes…Many of the professionals we encounter easily pass both tests, and their off-the-cuff judgments deserve to be taken seriously.


In other words, if you have a lot of experience, and if you have good, quick feedback on mistakes, then your intuition is likely to be better than random chance.

This, I think, characterizes our admissions office. In any given admissions committee, decades and decades of admissions experience are directed towards examining a single applicant and all of the information – essays, interviews, letters of recommendation, awards from external experts – we have about them. In fact, I laughed a little at Kahneman’s reference to “true legends of instant diagnoses are common among physicians”, because McGreggor Crowley, who directs our admissions process, is a physician, and if there is anybody who is legendary for his ability to “diagnose” an applicant, it’s him.

And we have good, rapid feedback too. We meet most students we admit soon after at CPW. We then spend four (or more) years living with them. They work in our offices. We advise them academically. We become friends as the years go on. So we don’t just have feedback on our decisions. We quite literally live with them.

Finally, there is the point that David made in his last blog post, which is essentially that there are many types of admissions processes, and that it doesn’t matter whether they are “fair” as much as it matters that they “work”, which is to say that they produce the sort of community that you aspire to be a part of.

I think there is a lot of truth in that. Fundamentally an admissions process is measured not by what it is but by what it does, which is of course to constitute a community. That doesn’t mean we aren’t reflective or analytical about the way we do things: in fact, we employ two terrific statisticians within our office alone specifically to run the data and tell us how to do things better!

But it does mean that the only real standard which matters is whether the students, the faculty, and the rest of the world think that MIT students are awesome people who do awesome things, and that our students feel at home here. By this standard, I think our process does a very, very good job.

And that, my friends, is no illusion.

15 responses to “The Illusion Of Validity”

  1. Yet another amazing post! smile

  2. Mark F says:

    Just started reading “Thinking, Fast and Slow” and fell in love with Kahneman, behavioral psychology, and cognitive biases. This is a great response I very much agree with and find representative of the MIT Admissions department. It’s also easy to take the MIT admissions for granted, that it generally does a great job in ways most (if not all) other university admissions departments don’t.

  3. m_quinn says:

    @Chris P

    Well, yadda, yadda, yadda: it’s the outcome of your admissions policies which tell the tale Chris. No matter how you sugar coat it; no matter what “scientific” methods you claim to employ, the sad truth is that you are operating an unfair and discriminatory admissions process.

    I guess MIT is going to have to decide what kind of institution it’s going to be:

    (1) A “JABBA the HUT” style institution – aloof, disconnected, take-it-or-leave-it, like Federal grant money but not your kids sort of college

    (2) A “reach out and serve your country” institution which actively seeks to inject some of that “intellectual fire power” (which – my opinion – only MIT can) into the hinterland.

    What’s it going to be????


  4. @m_quinn

    After every brilliant post, thanks for providing some hilarious entertainment! You never will stop trolling, will you?

  5. MIT EC '85 says:

    I’m confident that the MIT admissions process does a better job than random chance if the bottom 1/3 of the applicant pool is first removed from consideration. I do suspect that if the process narrowed down the pool to the top 1/3, then random selection would create a class about as “good” as the one using the current methodology. Of course that depends on the definition and measurement of “good” – a rather nebulous concept.

    My experience as an EC leads me to believe that the MIT applicant pool is so strong that if MIT admitted 4000 applicants, there would be no dilution in the quality of the student body. I have certainly interviewed many more top-notch applicants who were rejected than who were admitted. Those rejected applicants had to settle for inferior educations at second-rate school such as Caltech, Harvard and Yale, but they were such impressive people that I am sure they will overcome those obstacles. smile

  6. nniuq_m says:


    The sort of institution that selects those who have higher aspirations than to troll admissions blogs as they’re posted. And gives those students the resources they need to meet those goals and make a constructive difference in the world.

    I guess you are going to have to decide what kind of person you’re going to be.

  7. Ashley says:

    And how exactly do you propose MIT decide differently?

  8. lol says:

    @m_quinn TROLOLOLOLOLO

  9. Chris Ong ('16)? says:


    I think the outcome of their admissions process is quite outstanding smile

  10. an '11 says:

    I don’t know if talking to people for a few minutes at CPW is enough to know if they are awesome, as you claim all admits are.

    I met a lot of people in 4 years at MIT and enough of them were not awesome that I think you could improve quite a bit. If I could give you feedback to help identify the less awesome people it would be that you find that too many girls, Asians, rich people, people from TJ, and people who were born or lived overseas but have a green card/US citizenship are awesome, and too few people from schools where people rarely get into a top college awesome (that might be a recruiting problem). I picked those groups because I think they are more likely to be unempathetic, too proud, mentally ill, or lacking a sense of humor, and hence not that awesome.

    Of course, we might just have different tastes in what is awesome–you recommended some burger places to a mutual friend and I didn’t like them so we definitely have different taste in burgers. IMHO Mr. Bartley is king in Boston.

  11. intleyes says:

    Although I haven’t posted for sometime, I have been reading the undergrad admissions posts since 2004. My son was an EA for the class of 2009. He did graduate Course 16 in 2009. This semester he returned to MIT for his graduate studies. Through the most recent posts by Chris and others I now understand, for the first time, how all this has happened. I feel that the admissions process looks at the fabric one has woven in their life. Does your small piece, flaws and all, match (in the broadest sense of the term) and more importantly, enhance the fabric of the huge global quilt that MIT has become? As a parent and human being, my son’s life at MIT has been and continues to be the most humbling experience of my life. Are you the fabric that becomes a seamless, but individually brilliant hue in the patchwork? Only those who are sewing the quilt as a consortium can decide that. Its their quilt, not ours.
    My best wishes to all applicants. You are not a statistic no matter how others may try to portray you.

  12. Piper '13 says:

    @an ’11 — If you haven’t gone to O’Sullivans, you haven’t lived.

  13. Nasser '17 says:

    Hey Chris there will always be sceptics , anything you write or try to explain will never please these people such as m_ quinn. Please IGNORE him.

  14. Anthony L. '15 says:

    (preface: I recognize that I’m falling hard for a case of <; here… but I can’t sleep anyway and maybe this will help)

    @an ’11: excuse me?

    I know plenty of girls, Asians, “rich” people (for some definition of rich), and people who were born overseas who are awesome, both at MIT and elsewhere (I can’t say about people from TJ or a more restrictive definition of “rich”, because I haven’t met such people). I also know plenty of girls, Asians, rich people, and internationals (as well as people who don’t belong in any of the categories you named), at MIT and elsewhere, that I personally don’t find that awesome, but I often find that, upon closer inspection, such people often have other things that are awesome about them even if they still aren’t my type of person.

    To generalize your personal opinions like that and then to pass it on as “feedback” to the admissions staff… I don’t even.

    It’s true that there are people I wouldn’t get along with here, and there are people not admitted who I think would’ve fit wonderfully and been very happy here. But it’s a fact that there are more qualified applicants than there is space, and to select from that pool through the pinhole lens of an application (even if they try, as they do, to capture as much as they can) is never going to be perfect.

    I think admissions does about as well as one can realistically hope for, given all this.

    Welcome to the real world.

  15. DK says:

    What michal1 is saying is that he believes he would have [had] a better chance of being admitted to MIT if the process was completely random. He is right, however, he is also arguing the validity of fairness. Would it be fair to simply randomly select the number of students needed to fill a class from a pool of all qualified students? If they were all equally qualified, sure. And it will definitely diminish the ability to game admissions. But it would only be fair if everybody plays the same game. Everyone thinks they are special. Most believe they honestly belong at MIT. That is fine, yet, it won’t be long before an applicant cries foul when their name is not pulled from the hat. “But…did you read my essay? Don’t you recognize how difficult it was for me to….?” Fortunately, MIT has been doing this for a very long time.

    Admissions knows what they are doing, and whether anyone agrees with it or not is
    not a base for contention. MIT does what works for MIT, just like other schools do what works for them, too.