The Case of the Missing Bullet Holes

Lockheed P-38 Lightning. My Father Built these Planes During WWII, and He Always Loved Them.

The year is 1943 and America is deep in the desperate struggle to save the world from tyranny. The United States had entered World War II a little more than a year earlier following the December 7, 1941 attack by the Japanese on Pearl Harbor.

As the United States ramped up the war effort the government decided that to win this fight, in addition to brave soldiers and superior equipment, they would also need help from the brightest minds in the world. This lead to the founding of the Statistical Research Group (SRG). The SRG was located in Manhattan, just a block away from Columbia University, and before long many of the best mathematicians in the world were working there.

How smart were the people working at the SRG? Milton Friedman, a future winner of the Nobel Prize in Economics, worked at the SRG and it was later joked that while he was there he was often only the fourth smartest person in the room.

And who was the smartest? That was widely recognized to be the brilliant mathematician Abraham Wald, a Jew who had fled Europe prior to the start of the war.

The Air Force came to Wald with a problem. Too many of their planes were being shot down so they wanted to add extra armor to the vulnerable parts of the planes. Too much armor would make the planes too heavy to fly properly, so they couldn’t add extra armor over the entire plane. They asked Wald to tell them how much extra armor to add to the parts of the planes that were being hit most often.

To help Wald they had collected statistics on the bullet holes in planes returning from combat. They presented him the following statistics:

 Section of Plane Bullet Holes Per Sq. Ft. 
Engine 1.11
Fuselage 1.73
Fuel System 1.55
Rest of Plan 1.80

Given these statistics, what would you tell the Air Force?

The answer Wald came up with surprised them. He instructed the Air Force to put the extra armor not where the bullet holes were, but were they weren’t – on the engines! And how did Wald come up with this answer. Simple. He considered the missing bullet holes.  

The Air Force had presented him with statistics on planes that had returned safely from combat. Wald recognized this as biased sample that told a distorted and incomplete story. There were also a lot of planes at the bottom of the ocean, and Wald correctly guessed that these planes were full of bullet holes in the engines.

The Air Force followed his advice and the results were stunning. Immediately more planes started returning safely from combat saving the lives of countless pilots and crew members.

Wald correctly identified this as not so much a math problem but as a problem of survivorship bias, and once you understand this concept you start seeing it everywhere. Survivorship bias tells a lot of distorted and incomplete stories and looking for the missing bullet holes, as Wald had done, can save you from making bad decisions based on inaccurate and incomplete information.    

Fortune Cookie Lottery Winners

On March 30, 2005 the Powerball lottery drawing produced 110 2nd prize winners who had picked 5 of the 6 numbers correctly. This entitled each of them to winnings of over $100,000. Lottery officials found this strange since, based on the number of tickets sold, there should have only been about 5 2nd prize winners.

Searching for an explanation the officials asked winners how they had picked their numbers. An article in The New York Times described what happened next:

“Our first winner came in and said it was a fortune cookie,” said Rebecca Paul, chief executive of the Tennessee Lottery. “The second winner came in and said it was a fortune cookie. The third winner came in and said it was a fortune cookie.”

The fortune cookie with the lucky numbers was eventually traced to a factory in Long Island City, owned by Wonton Food, which produces four million fortune cookies a day.

“That’s ours,” said Derrick Wong, of Wonton Food, when shown a picture of a winner’s cookie slip. “That’s very nice, 110 people won the lottery from the numbers.”

Reporting on the incident Ben Carlson wrote cynically, but more truthfully than we care to admit, “If any of these 110 people worked in the investment business they probably would have started their own newsletter – My Secret Formula for Winning the Lottery.”

And they could have made the newsletter sound very convincing. After all, the author could have pointed to109 other people who also followed this formula and won big. But you wouldn’t fall for it, because you would look for the missing bullet holes, and you understand that there are thousands of planes piloted by fortune-cookie-lottery-number-picking pilots laying at the bottom of the ocean.

Other Examples of Survivorship Bias

  • Many investing companies start new mutual funds in what are known as “fund incubators.” These funds, employing various strategies, are kept small and are not available for public investing. After several years in the incubator the most successful ones are rolled out to the public touting their stellar investment records. The underperformers are discontinued in an attempt to keep the missing bullet holes hidden forever.
  • In the classic book The Millionaire Next Door authors Thomas J. Stanley and William D. Danko state that business owners make up the largest group of millionaires next door. But this is not the same as saying that most business owners are millionaires. Most businesses fail and the ocean floor is littered with planes piloted by brave but ultimately unsuccessful entrepreneurs. So before jumping in and starting your own business, consider the missing bullet holes.
  • My wife and daughter love the real estate shows on HGTV. These shows make flipping houses look effortless and exciting. And there is no shortage of books and infomercials telling you how easy it is to make money buying real estate with little or no money down. While it is certainly possible to make money in real estate, it is not easy. Don’t forget to look for the missing bullet holes. They are not difficult to find. Lots of would-be real estate tycoons, including personal finance radio guru Dave Ramsey, have crashed in spectacular fiery fashion. Before you start flipping houses, consider the missing bullet holes.

My purpose in discussing survivorship bias is not to talk you out of taking risks. Indeed, I believe that anything worth doing involves risk. I just don’t think you should take risks based on the distorted and incomplete picture that survivorship bias presents.

Speaking of this Ben Carlson said, “Every inspirational speech by someone successful should have to start with a disclaimer about survivorship bias.”

Instead, clear up the picture first by searching out the missing bullet holes and seeing what you can learn from them. They often tell the most interesting and useful part of the story. This won’t guarantee you success but it will increase your chances for a safe return by allowing you to put some extra armor on the vulnerable areas of your plane. After doing that, buckle your seat belt and prepare for takeoff!

Note: I read the story of the missing bullet holes in the excellent book How Not to Be Wrong: The Power of Mathematical Thinking, by Jordan Ellenberg.

  5 comments for “The Case of the Missing Bullet Holes

  1. TheRetirementManifesto
    November 8, 2017 at 7:17 am

    Wow, fascinating story, and a perfect analogy for using survivorship bias to seek out the missing bullet holes! Congrats on the Rockstar award today, well earned!

    • Brent Esplin
      November 8, 2017 at 7:46 am

      Thanks, Fritz. I really do love the story of the missing bullet holes. It has a lot of important lessons in it.

  2. November 8, 2017 at 11:21 am

    I see this all the time, everywhere! Great story.

    • Brent Esplin
      November 8, 2017 at 11:31 am

      I do also. Once you learn the concept you start seeing it everywhere. That is why it is so important to understand. Glad you enjoyed the story.

Leave a Reply