What Analysts Really Do: Judgement, Not Just Data
In a world awash with algorithms and dashboards, it’s easy to forget that the heart of analysis lies not in answers, but in asking the right questions. This piece revisits economist Alain Enthoven’s enduring principles — reminding us that true insight begins where data ends and judgment takes over.
In a time when data-driven has become the cri de cœur, the almost sacred cow in both boardrooms and policy groups alike, it is extraordinarily simple to confuse the complexifying of information with actual insight. We are drowning in dashboards, predictive models, and algorithms that offer the best way ahead.
And yet, as Alain C. Enthoven, giant of policy analysis, one of the U.S. Department of Defence’s original Cold War ‘whiz kids’ and longtime RAND resident, wisely explained over many decades ago, the proper aim of policy and program analysis is not to mechanically regurgitate an optimal solution, but something much more subtle, and, well, human. No less relevant today than when they were first written in 1974 and then revised, his Ten Practical Principles seem downright prescient in the face of our continuing struggles with complexity on an entirely new level.
Process > Solution
One of the most powerful reminders that Enthoven gave is that good analysis is in the service of judgment, not in the replacement of it. This is not merely semantic, it is a deep criticism of the current idea enough data thrown into a fancy model will automatically show the “optimal” course of action. A responsible executive should, and must, reserve the final right of judgment in the murky world of public policy where values, uncertainties, and intangibles often tower.
The job of the analyst then is not that of an oracle, but instead to cast a light upon the manner in which the decision depends. These include those all-important judgments – to seek and bring to the surface the great questions of value, the obscure doubts, and the non-measurable factors, instead of burying them beneath a mass of figures. The greater value – even more than the ultimate recommendation (which is seldom general enough to be rigorously defended, in any case) – may be more in the process: the framing of the problem, the alternatives generated, the data analyzed, and the criteria used.
Enthoven cautions many conventional decision theorists are misled in seeking the most suitable answer, based on assumptions, instead of revealing the reliance of answers on the assumptions. It is not just a question of adjusting parameters, but rather of solid sensitivity analyses and break-even calculations that will show what really counts and what does not. In his succinct wording, there is hardly ever a best solution, yet you do not need one to recognize, and, more importantly, prevent bad ones.
Check Your Facts, Present Them Simply
Getting the bare facts roughly correct may be the most underestimated, and yet the most vital step to tackling any problem. We are widely taught to process data, not as much to test its validity. A skeptical – almost forensic – look at data involves the ability to pose correct and often difficult questions. It consists of being aware of the exact definitions and assumptions that go into shaping the data you receive. It requires a high degree of triangulation, i.e. coming up with sources of information that are independent and often mutually exclusive.
And, most importantly, it involves seeking contradictions, comparing and contrasting numbers, and questioning the motives and conflict of interest of the people presenting the information. Are they painting a picture of improvements because it is so, or because their livelihood is on the line? An unpleasant idea, yet one which must occur to any serious analyst.
But the anecdote that rings true is surely then U.S. Secretary of Defence Robert McNamara complaining he wanted a three-page memo on the perfidious antiballistic missile defense system, having been presented with a forty-page disquisition by Enthoven. The lesson? Make it simple. You may thrash through maze-like complications to get at fundamental findings, but your work is not finished until you can put the fundamentals in simple terms to the overworked decision-maker.
Unless you can distill it, you have not really understood it yourself. This simplicity is not the matter of oversimplification, it is the matter of deep clarity, of concentration of the user on assumptions and results rather than on complex computations.
Culture, History, and the Grand Total
More than ever, perhaps, modern analysis runs the danger of being tunnel-visioned. A powerful antidote proposed by Enthoven: always begin by examining the big totals. Don’t merely study the Polaris missile program as itself; study it in the context of the whole strategic offensive force. Apply the same to pollution control: does what appears to be a solution merely relocate the waste elsewhere? Or in healthcare, where hospital behaviour often requires consideration of the larger ecosystem of doctors and patients rather than just autonomous units.
Besides, efficient analysis is often not a sterile, de novo undertaking. It requires a profound look into the applicable history – why are there still problems, what policies have already been tried and failed, what can the experience of previous attempts teach us? And most importantly, it needs an understanding of the cultures of the organizations in question – their common beliefs, objectives, and ways of behaving.
An analyst can design the most beautiful solution on paper but when it conflicts with the incentives or the long-practiced mode of operation of the implementing organizations, it is bound to fall apart.
Even the most beautiful analyses, backed by ideal data, becomes useless, as long as it cannot be effectively implemented. This implies doubting, and usually painfully, evaluating whether individuals and institutions will really react as expected, and whether the plan to be taken is consistent with the institutions to execute it.
Analysis, particularly policy analysis, is then not only a matter of discovering answers to existing questions, but of creating new alternatives and, more fundamentally, of discovering the questions worth asking in the first place. The process is iterative and includes continuous learning – requiring a progressive clarification of meaning that requires humility, discipline, and an unblinking attention to the cruddy, human reality of how policy is actually implemented.
In an age where we are all too often carried away by data, the principles put forward by Enthoven help us to remember that human judgment remains the source of wisdom, guided but never superseded by analysis.
Read ‘Ten Practical Principles for Policy and Program Analysis’ by Alain C. Enthoven (RAND) here.