Politi-Score: Dumb Politics

Recently there’s been some hubbub now that more notable forums are trying their hand at crunching PolitiFact’s rulings, and arriving at similar conclusions, with a piece by one Eric Ostermeier at a website called “Smart Politics” which is part of the University of Minnesota website. (U of M???? I prefer the Wolverine-type, actually.)

The title of Ostermeier’s article is, “Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats.” Looking over his tables, which are presented in reverse with Pants on Fire first, I was immediately struck by the fact that the 511 total number wasn’t enough--even if he was just counting “Current and Former Political Officials.”

A Smart Politics content analysis of more than 500 PolitiFact stories from January 2010 through January 2011 finds that current and former Republican officeholders have been assigned substantially harsher grades by the news organization than their Democratic counterparts.

Ostermeier supposedly tallied these 511 “statements put through the Truth-o-Meter test from January 1, 2010 through January 31, 2011.” That’s 13 months. Therefore, my educated guess is that he only counted the rulings of PolitiFact National.

Ostermeier segregates the rulings similarly to the method I have followed: the “current and former political officials” from those of “pundits, groups, emails, blogs and miscellaneous” rulings.  It appears that he wanted to determine if the pundits were graded more “harshly” given they would not be driven by voter appeal. But his conclusions are clear on the “current and former political officials”:

Republican statements were graded in the dreaded "false" and "pants on fire" categories 39 percent of the time, compared to just 12 percent for statements made by Democrats.

That means a supermajority of falsehoods documented by PolitiFact over the last year - 76 percent - were attributed to Republicans, with just 22 percent of such statements coming from Democrats.

As a consequence, Democrats have therefore been presented as much more truthful - with over 75 percent of statements receiving the top three grades of True (16 percent), Mostly True (27 percent), or Half True (33 percent).

Less than half of Republican statements graded by PolitiFact were regarded as half truths or better - just 90 out of 191 (47 percent).

Now that he’s given me his breakdown, I can compare it with mine.  First my data needed to be filtered to PolitiFact National.  Secondly,  the pundits, orgs, emails, etc., were separated out so my measure would match his criteria.   The comparison reflects two different periods of time:  Ostermeier's is for the 2010 calendar year, while mine is for PolitiFact rulings since its inception in 2007 through the end of 2010.

Relative to Ostermeier's calculations, the  Politi-Score proved to be a useful tool to show that compared to my much larger sample (indeed, almost the entire population), his sample scored a 55 for the Democrats compared to my 56, and he had a 36 for the Republicans, whereas my sample gave them a 44, closing the gap between the Republicans and Democrats almost 37% closer than what Ostermeier had calculated.  The Democrats had 70% of statements rated Half True and above (compared to his 75%), while the Republicans had 54% of statements rated Half True and above (compared to his 47%).

The False and Pants on Fire rulings were closer as well, although still favoring the Democrats. The Republicans held 63% of the False and Pants on Fire Statements, as opposed to 76% per Ostermeier's calculations. In addition, the Republicans were ruled False and Pants on Fire for 31% of their statements (compared to Ostermeier's 39%) and the Democrats 17% of their statements (compared to Ostermeier's 12%).

My measure for those who were not former or current politicians—the pundits, e-mails, blogs, groups, etc.—appear not to coincide with Ostermeier's claim that:

But this potential selection bias - if there is one at PolitiFact - seems to be aimed more at Republican officeholders than conservatives per se.

An examination of the more than 80 statements PolitiFact graded over the past 13 months by ideological groups and individuals who have not held elective office, conservatives only received slightly harsher ratings than liberals.

According to my larger sample, non-office holders for PolitiFact National scored a 45 for the Politi-Score for Democrats/Liberals (versus 55 for office-holders) and 25 for Republicans/Conservatives (versus 44). So the Democrats/Liberal pundits were about 25% lower, the Republicans/ Conservative pundits were a whopping 80% lower. The pattern held with PolitiFact ALL non-office holders as well: the Democrats fell a bit to a Politi-Score of 44 while the Republicans increased a bit to 27. In other words, conservatives pundits and organizations not only received far harsher rulings than office-holders, but received greater than 60% more harsher ratings than liberals, which is not exactly “slightly”.

Lastly, my conservative counterpart PolitiFact critiquer Bryan White had this to say about the False and Pants on Fire rulings:

I would again call attention to the divide between the "False" and "Pants on Fire" ratings. PolitiFact to date has offered no metric for objectively distinguishing one from the other. This in turn strongly suggests that subjective judgment serves as the ultimate criterion for grading a statement as "Pants on Fire."

If it’s easier to ascertain a “False” as opposed to “Pants on Fire” what would happen if all those Pants on Fire rulings were “offered the same metric” of a False? After all, Pants on Fire is just a “subjective” interpretation of False….would it change the rulings overall? The answer is it would raise the Politi-Score averages for both Democrats and Republicans, and obviously, the Democrats would still hold the edge:


Note both "all" Dems and Repubs are a bit closer than Smart Politics' calculations with a False + Pants on Fire.

 As for the theory of the Truth-o-Meter’s ratings being the result of “selection bias” as opposed to the appearance of a “measure of truthfulness” the PolitiFact partnerships might provide an answer: for example, PolitiFact Wisconsin’s parent company is the owner of conservative talk radio in Milwaukee, leading one blogger to allege conservative bias, and the Politi-Score showed the Democrats with just the tiniest of leads over the Republicans during the 2010 elections, the Democrats averaging just over 23 and the Republicans just under 23 on the Politi-Score, very “barely true” scores with almost no variance between them, unlike PolitiFact Florida where the Republican Politi-Score is less than half that of the Democrats. PolitiFact Virginia’s paper has Eric Cantor’s wife on its board and endorsed McCain in 2008 and Bush in 2004, but their Politi-Score for office-holders is 31 for Democrats and an even more dismal 19 for Republicans (and during the elections, the Republican Politi-Score exceeded the Democrats as well). While there’s a website dedicated to the alleged liberal bias of PolitiFact Texas, an analysis of their False and Pants on Fire rulings shows the Democrats slightly edging out the Republicans, with 30% to the Republicans’ 29%. The PF Texas Politi-Score shows them closer as well: Democrats have a 44 average versus Republicans' 40.

In other words, there appears to be a trend of bias, whether it’s selection bias or rating bias...and the evidence clearly indicates that the rulings have been more favorable to the Republicans outside PolitiFact National.  This is borne out by filtering out of my data the rulings for PolitiFact National and Florida:  without the St. Petersburg crew, the Democrats' Politi-Score is 42 compared to the Republicans 36, so the Democrats are still higher on average.  They both have 10% of rulings rated Pants on Fire in lieu of the lopsided variance at PolitiFact National/Florida for this category; the Republicans have 33% of rulings False and Pants on Fire to the Democrats 29%, and the Republicans had 57% of all the False and Pants on Fire rulings awarded to both parties.

In a previous post, I had provided some data relative to statements rated by FactCheck.org as compared to PolitiFact. They rate many of the same statements, although they do not offer a grading system like PolitiFact.  Factcheck also tends to focus on statements they believe to be untrue: my study of six months of their reviews found that 85% of the statements they rate are found to be either false or misleading. They also rate more Republicans and find more of their statements false. While this is not to condone PolitiFact, the rating system it employs may serve to cause it to be taken less seriously because it is often fairly easy to discredit such a specific rating under either ideology, especially an ambiguous one like “Half True”. They may be better off not using such a system at all, or using one similar to Snopes.com which has loop hole categories of “Undetermined” and “Unclassifiable Veracity” to minimize the allegations of bias.

Ostermeier (according to Bryan White) does "point[] out  that PolitiFact's  presentation  encourages readers to use the ratings to draw conclusions about the subjects receiving the ratings…” To me, ratings such as these beg gauging, and that is the risk inherent in such a system. PolitiFact may end up being the “FactCheck Light” of such websites because of it...even with Liberals.

Postscript:  Here's even more analysis of Ostermeier's stats.