Monday, May 23, 2022

Wharton professor Philip Tetlock and journalist Dan Gardner on how you can improve your ability to predict

Here’s a question that definitely was not asked in the forecasting tournament: How many piano tuners are there in Chicago?

Don’t even think about letting Google find the answer for you. The Italian American physicist Enrico Fermi-a central figure in the invention of the atomic bomb-concocted this little brainteaser decades before the invention of the Internet. And Fermi’s students did not have the Chicago yellow pages at hand. They had nothing. And yet Fermi expected them to come up with a reasonably accurate estimate.

Outside Fermi’s classroom, most people would frown, roll their eyes, scratch an ear, and sigh. “Well, maybe”-long pause-and they would offer a number. How did they arrive at that number? Ask them and they would shrug and say nothing more informative than “It seems about right.” The number came out of a black box. They had no idea how it was generated.

Fermi knew people could do much better and the key was to break down the question with more questions like “What would have to be true for this to happen?” Here, we can break the question down by asking, “What information would allow me to answer the question?”

So what would we need to know to calculate the number of piano tuners in Chicago? Well, the number of piano tuners depends on how much piano-tuning work there is and how much work it takes to employ one piano tuner. So I could nail this question if I knew four facts:

1. The number of pianos in Chicago
2. How often pianos are tuned each year
3. How long it takes to tune a piano
4. How many hours a year the average piano tuner works

With the first three facts, I can figure out the total amount of piano-tuning work in Chicago. Then I can divide it by the last and just like that, I’ll have a pretty good sense of how many piano tuners there are in Chicago.

But I don’t have any of that information! So you may think I’ve wasted my time by exchanging one question I can’t answer for four.

Not so. What Fermi understood is that by breaking down the question, we can better separate the knowable and the unknowable. So guessing- pulling a number out of the black box-isn’t eliminated. But we have brought our guessing process out into the light of day where we can inspect it. And the net result tends to be a more accurate estimate than whatever number happened to pop out of the black box when we first read the question.

Of course, all this means we have to overcome our deep-rooted fear of looking dumb. Fermi-izing dares us to be wrong. in that spirit, I’ll take my best shot at each of the four items:

1. How many pianos are there in Chicago? I have no idea. But just as I broke down the first question, I can break this down by asking what I would need to know in order to answer it.

a. How many people are there in Chicago? I’m not sure, but I do know Chicago is the third-largest American city after New York and Los Angeles. And I think LA has 4 million people or so. That’s helpful. To narrow this down, Fermi would advise setting a confidence interval –a range that you are 90% sure contains the right answer. So I’m pretty sure Chicago has more than, say, 1.5 million people. And I’m pretty sure it has fewer than 3.5 million people. But where is the correct answer within that range? I’m not sure. So I’ll take the midpoint and guess that Chicago has 2.5 million people.

b. What percentage of people own a piano? Pianos are too expensive for most families-and most who can afford one don’t really want one. So I’ll put it at one in one hundred. That’s mostly a black-box guess but it’s the best I can do.

c. How many institutions-schools, concert halls, bars-own pianos? Again, I don’t know. But many would, and some, like music school, would own many pianos. I’ll again make a black-box guess and say that it’s enough to double the per person number of pianos to roughly two in one hundred.

d. With those guesses, I can do some simple math and conclude that there are fifty thousand pianos in Chicago.

2. How often are pianos tuned? Maybe once a year. That strikes me as reasonable. Why? I don’t know. It’s another black-box guess.

3. How long does it take to tune a piano? I’ll say two hours. Again, it’s a black-box guess.

4. How many hours a year does the average piano tuner work?

This one I can break down.

a. The standard American workweek is 40 hours, minus two weeks of vacation. I don’t see any reason why piano tuners would be different. So I’ll multiply 40 hours by 50 weeks to come up with 2,000 hours a year.

b. But piano tuners have to spend some of that time traveling between pianos, so I should reduce my total by that much. How much time do they spend between jobs? I’ll guess 20% of their work hours. So I conclude that the average piano tuner works 1,600 hours a year.

Now I’ll assemble my guesses to make a final calculation: If 50,000 pianos need tuning once a year, and it takes 2 hours to tune one piano, that’s 100,000 total piano-tuning hours. Divide that by the annual number of hours worked by one piano tuner and you get 62.5 piano tuners in Chicago.

So I will estimate that there are sixty-three piano tuners in Chicago.

How close am I? Many people have taken a crack at Fermi’s classic puzzler over the years, including the psychologist Daniel Levitin, whose presentation I’ve adapted here. Levitin found eight-three listings for piano tuners in Chicago yellow pages, but many were duplicates, such as businesses with more one phone number. So the precise number isn’t certain. But my estimate, which rests on a lot of crude guesswork, looks surprisingly close to the mark.