The Inherent Limitations of Statistics

In many areas of our culture and lives, statistics have assumed outsized importance. As someone who has studied, used and marketed statistics across a number of industries, this development seems a pendulum swing too far to one side; the encouragement from media and partisans to see statistics as conclusive and as fact.

As I’ve discussed previously, facts are often misleading or irrelevant. The same rule applies to statistics (a subset of the fact universe), where context is perhaps more important than for other statements. One example exists in the recent news stories pertaining to the economic devastation from the pandemic. The reports today of an astonishing 22 million Americans have filed for unemployment over the past three weeks is a tragic reflection of the times, and is widely noted as a multiple of prior “worst” periods of our history… but is the picture painted an accurate one, particularly as regards those comparisons?

The Current Economic Downturn

In recognition of the unique nature of the current economic downturn, Congress has broadly expanded the qualifications for who qualifies for unemployment. Included in the present figures are the self-employed, shorter-term employees, “gig” employees and contractors, and other categories not usually included. Reflecting the forced closure of so many related industries — particularly hospitality and transportation — the consensus is that these categories comprise the vast majority of new filings. Actual numbers are not readily available, leading to even less utility for the provided statistics; we are given an irrelevant comparison, the proverbial apples, and oranges, and have little basis for understanding. In addition, there is a significant question as to the accuracy of the numbers given that the majority of states have been unable to keep up with the submissions; it is possible that those numbers, as large as they are, remain a serious underestimation.

This is not to question the powerful trauma being inflicted on our economy or our nation; these are real people in dire circumstances, and the legislation appropriately recognizes and addresses that… but in our evaluation of the scope of our response, we are given a picture that is at best incomplete, and at worst misleading. The impact on our economy, the challenges of support and repair, demand difficult choices in the allocation of unprecedented assets; understanding the components clearly and in context is critical.

An example: let’s suppose that the actual figures broke down as follows: of the 22 million, 21 million reflected those workers usually excluded from the unemployment rolls. Would the response differ? Should the proper payments be made in lump sums, or for different durations than those for conventional recipients? Should the response include work programs that reflect the youth and more general training of that group? And what would it say if that constituency represented only 7 million of the 22 million? That would demand a far greater response to the corporations above them, and reflect a true depression-style challenge. The difference between those underlying numbers is enormous, and yet not provided in the various media reporting.

Similar issues related to multiple facets of the pandemic: the recorded infected totals reflect not actual numbers with the virus, but a percentage of those who have been tested, a potentially substantial variable. The number of fatalities related to the definition of a COVID-related death: many who die of potentially the virus are not tested, and therefore excluded, while others who are included may have died from a variety of issues. Virtually every person who dies can be said to have died of heart failure since the ultimate determinate is that their heart stopped beating… it is the reason for that tragic event that is important. As a result, almost all of the broadcast statistics regarding the pandemic are questionable and incomplete.

In my experience, statistics — well defined and in context — can be incredibly useful in framing the questions and challenges that need to be addressed. In and of themselves, however, statistics tend to be far less useful in providing actionable answers and solutions. For statistics to have real meaning, we need to understand their origin, their construction, and their relevancy to the issue at hand, three critical components that we are rarely offered… even then, we should use them to open important discussions, not to close them.