In "Expert Political Judgment: How Good is It? How Can We Know?", Philip E. Tetlock presents empirical evidence that political pundits are overwhelmingly wrong in their predictions about future political events (book, interview). What's more, he claims that the more knowledgeable an expert is in some particular field, the worse his/her predictions about that field. Tetlock uses Isaiah Berlin's famous take on Archilochus' equally famous fragment ("The fox knows many things, but the hedgehog knows one big thing") to analyze the situation.
According to Berlin, thinkers can be classified as either "hedgehogs" or "foxes". The hedgehogs are those who "relate everything to a single central vision, one system less or more coherent or articulate, in terms of which they understand, think and feel" and the foxes are those who "pursue many ends, often unrelated and even contradictory, connected, if at all, only in some de facto way". According to Tetlock, political pundits are generally hedgehogs, and so understand the political world through whatever more or less coherent system (liberalism, conservatism, etc.) they endorse. Consequently, they tend to over-emphasize evidence that supports their worldview and under-emphasize evidence that runs contrary to it. In essence, Tetlock argues that political pundits are bad predictors because the nature of punditry is such that it causes the pundit to be a bad judge of evidence. (Notice that this is an argument about how evidence is evaluated, not about the peculiar nature of prediction, the unknown future, etc.)
No doubt, this make for great cocktail party banter. But does it mean anything for historians of philosophy?
I think that it might. After all, aren't most of us hedgehogs? Our individual work usually revolves around a small set of philosophers, if not a small set of philosophical questions with which a small set of philosophers was concerned, and each of us usually has a “line” that s/he is trying to push. I, for one, can confess: my dissertation (on which I'm currently working) is built on a single idea, and the whole work is an attempt to articulate a more or less coherent system of historical facts and philosophical arguments that explicate this idea. Pretty hedgehogish, don't you think?
Now here's the troubling part. If Tetlock is right, then whether I am consciously aware of it or not, my supposed expertise is causing me to misjudge the historical evidence. Moreover, if I am not the only hedgehog, then maybe this problem is prevasive! Does the field of history of philosophy have any hope of achieving a relatively accurate historical understanding of the development of philosophy? It seems that empirical evidence is against us.
Perhaps we need to wait for studies that somehow average pundit opinion. Maybe the sum of us do better than each taken separately?…. Any takers for this project?
02 January 2006
Hedgehogs and Historians
Subscribe to:
Post Comments (Atom)
5 comments:
This is pretty interesting. But there might be some relevant differences between the historian and the pundit (average yearly income being the major one...).
For instance, one potential difference is the goal of the historian versus that of the pundit: perhaps the historian is trying to understand (in some suitably worked out sense of that word) whereas the pundit is trying to predict. That is, explanation and prediction are not the same thing. Early modern philosophers were hedgehogs, at least in intent, and so maybe we must be charitable and try to reconstruct early modern philosophy in hedgehog terms.
Then again, its probably true that the early modern philosophers, despite their intentions, were inconsistent and incomplete, so if we, in attempting to understand them, impose a false homogoneity and consistency on them, maybe we should just give up and move somewhere warm.
No individual endeavor can completely and truthfully capture everything there is to know about anything, let alone history. Some things are simply lost to time. Other things are idiosyncratic and fail to be explicable. At some point, a reconstruction will have to fall back on some sort of theoretical "filling in" by the historian. That's fine. That's all we can expect.
Now, the filling in might be right or wrong, but the best the historian can do is make a good argument for his or her point of view. So the project becomes marshalling evidence and presenting reasoned, convincing arguments that the point of view is sound. Proof and completeness are not to be expected. And the historian's arguments are to be judged on the merits and strength of his or her arguments (not, as the previous poster bizarrely suggests, on how well they emulate the arguments of those under study).
Being a hedgehog helps the historian construct convincing arguments. The more we know about a particular subject, the better we are able to fill in the missing details. But being a fox is also helpful, since it leads to information about the context of a subject, and this helps fill in based on what might be expected within that context.
In any case, it's reason that's the ultimate arbiter, not any particular predictive power or absolute correctness. (And, by the way, we can approach a fuller understanding of the past through the work of many, even competing, hedgehogs. So push your line as far as it will go. Posterity will see how much it stands to reason.)
Let me respond to only one of your points, bg:
Although prediction and explanation are indeed very different, Tetlock's analysis of pundits' failure did not appeal to the fact that they failed *in predicting*. His analysis only concerned how one evaluates evidence. It seems to me that whether the evidence is used in generating predictions or providing fodder for explanations the same problem obtains. Namely, a hedgehog will skew the evidence.
And, by the way, I second the motion for moving to somewhere warm.
And, DMM, thanks for the confidence that I should push my line as far as it will go (which isn't that far!).
I agree with you that the game is about "marshalling evidence and presenting reasoned, convincing arguments that the point of view is sound. Proof and completeness are not to be expected." However, I think Tetlock research suggests that at hedgehogish levels reason itself, in a way, breaks down. Hegehogs, as an empirical matter, skew the evidence and thus present flawed arguments. In theory, the game is about making reasonable arguments. In practice, Tetlock research suggests, the game consists of making biased arguments.
However, I do take your point that when evidence is missing (as it always is in historical matters, but also with regards to the contemporary political sphere), a hedgehog may be able to interpolate better than a fox. This is a valuable skill, and indeed a required one when it comes to telling historical tales. But I think the problem still remains: although a hedgehog may be able to interpolate better than a non-hedgehog, his/her interpolation will be less accurate if the data set from which s/he is interpolating is skewed.
So let the arguments be biased. Let the data set be biased. I still don't think this is an individual enterprise, so it's fine if one person rocks the boat. Someone else will eventually rock it the other way.
To draw the analogy to the accuracy/precision distinction: You're worried that an individual's work will be precise but not accurate. That the data points within his or her work will coalesce around a false measure. I'm arguing that this shouldn't be evaluated within the work of individuals, but across the historical project as a whole. If we take the work of many individuals as the data points we're worried about, then our history - as a whole - will be imprecise, but, I think (pray?), accurate. In any case, I think this implies that accuracy should not be the only standard by which we measure our own work. Precision - i.e., good argument - should be an equally important consideration.
Of course, historians should try to "get things right." But an inability to be sure that things are right should not paralyze us. If we're wrong, someone will eventually correct us.
I think this is a reasonable argument. I intentionally left open this sort of hope in my original post, but, alas, I am aware of no empirical evidence, akin to Tetlock's, about it.
Post a Comment