A college student by the name of Peter William Hurford published an interesting review with his own PolitiFact (PF) Politi-Score/Truth Index calculations. He titled it “Rating Presidential Candidates by the Polifact Heuristic.” While he’s spelled PolitiFact badly as Polifact, I’ve rarely heard the term heuristic. Anyway, Hurford adds the same type of graphics, although his analysis isn’t as sophisticated as my Truth Index/Politi-Score counterpart Steve at Quibbling Potatoes.
In “mass communication” a heuristic is defined this way:
In the study of media effects, judgmental heuristics have been shown to play an active role in the simplifying of news and political communication. Use of these cues and other signals from elites allow average people the opportunity to achieve a modest level of rationality in reaching a decision. This can be accomplished without having to devote any significant measure of cognitive effort normally required to arrive at thoughtful and considered choices. The limited capacity theory model and other information processing models have been influential in the study of how people encode, store and retrieve political information. Most people maintain a minimum level of interest in public affairs, and therefore employ simplifying shortcuts to arrive at political judgments. Common examples include referring to the complex military and intelligence activities by NATO forces in the Middle East simply as “the war on terror,” a reversal of a specific policy or position as a “flip-flop,” and the homogenization of any type of broad government assistance program as “socialism.”
In other words, PolitiFact acts as a “cue” or “signal” with its Truth-o-Meter (and its popular brand “Pants on Fire”) to simplify news and political communication.
What Hurford does is basically work up to the weighted average as employed by the Truth Index, only assigning double the value of the Truth Index: instead of 150 points for Pants on Fire, or 1.5, Hurford assigns 300 points or 3.0, by 100-point (or 1.0) increments between each instead of 50 points (or .5). He starts with some of the presidential candidates and looks at the False and Pants on Fire as a percentage, then looks at the True and Mostly True as a percentage, and then adds other political figures, columnists and pundits who’ve been frequently rated by PolitiFact. He finally gets into a weighted average assigning his own scores (as described above).
First, comparisons may be invalid because of statement sample biases. Again, it may be that 90% of every statement made by Glenn Beck is trustworthy and accurate, it’s just that of the specifically interesting things he says, his statements appear to be largely untrustworthy and inaccurate.
Second, comparisons may be invalid because of person inclusion biases. It certainly does look like Democrats appear seem to be more truthful than Republicans, but this probably isn’t true because of the large number of currently campaigning Republicans placed next to the large number of non-campaigning Democrats, and the propensity of people to utter falsehoods while under the pressures of the campaign trail and not in other locations.
Third, generalizations may be invalid because of an insufficient sample. It certainly looks like columnists (George Will and Paul Krugman) are far more trustworthy than pundits (Bill O’Reilly, Glenn Beck, and Rachel Maddow) and politicians (nearly everyone else), but the fact that we only included two famous columnists out of hundreds might mean that we have an unrepresentative sample, and columnists are largely inaccurate.
So what does this add up to?
When we acknowledge that we’re using a heuristic, and thus may not have accurate results but instead results that we consider accurate enough, we might not want to consider supporting or voting for Michelle Bachmann or Herman Cain on the basis of their continual habit of lying or being misinformed.
 |
| Hurford's "Politi-Scoring" |
At the left is the weighted average "Truth-Score" Hurford came up with for individual politicians/ pundits/ columnists. The results are somewhat similar to mine. He lists them least truthful to most, with the names in red font being the Republicans, the blue Democrats. At the bottom of this post I've put my most recent individual 2011 Truth Index calculation for comparison. There's also one I did previously encompassing all PF rulings, not just those for 2011.
My conservative counterpart begs to differ, of course, in that he believes Hurford has made the error of “trusting elite opinions.” As I have gathered PF’s rulings statistics, I’ve made it a rule to qualify individual truth indexes by noting that a few rulings don’t mean a lot—what Hurford would term “an insufficient sample”. My last individual ruling rating was for those with 15 rulings or more, because I also believe that any conclusions reached should be based on “results that we consider accurate enough.” In addition, comparing those rulings to similar rulings done by other fact-checkers for the same statements provides a certain amount of validation to the accuracy of those results. All biases are not equal. That's the reason for what I call the "Politi-Score."
Finally, the PolitiFact Truth-o-Meter rulings as quantified offer other statistical categorical “heuristics” beyond individual on a macro level, such as those critiqued in this blog and by Steve at Quibbling Potatoes: calculating each PolitiFact state’s heuristic average, or a local versus national heuristic, one by subject, office-holder versus non-office-holder or month-to-month as related to those political variables which might be affecting it. In other words, while individual rankings may be where the most interest is, we shouldn't consider them until there IS a sufficient sample to do so. Given the quantity of rulings, these other areas of interest can be looked at in the mean time.
 |
| This chart only includes 2011 PF rulings on politicians (Rick Perry would be lower if ALL his rulings [pre 2011] were included, so would more closely approximate Hurford's "heuristic" ratings). |
No comments:
Post a Comment