Yes, Virginia, PolitiFact isn't a Bunch of Social Scientists

Still No Real Proof of PolitiFact Bias from PolitiFact Bias

Well, I guess all the relentless pressure worked (ha ha!). After six months of nothing on PolitiFact Bias’ Research page, we finally have that all-encompassing, definitive, proof-positive, documented study to end all studies, that gives us evidence beyond a shadow of a doubt, and even includes a measure, of PolitiFact’s liberal bias.

That's all just sarcasm, of course,  because after downloading and reading the 6-page report, it reminded me of another “study” which was posted last month, a fact-check by PolitiFact New Jersey, about a liberal blog called Uppity Wisconsin which had added up all the Trues and Mostly Trues of several governors and concluded that that since there were less for Republicans compared to Democrats, Republicans must lie more (which made it all the way to a claim using the data by a New Jersey state representative, which was then picked up by PolitiFact New Jersey).

But PolitiFact Bias’ methodology was a bit more sophisticated, and went the other way on the Truth-o-Meter. It added up all the Falses and Pants on Fire from PolitiFact’s beginning through the end of 2011, then took a percentage of the Pants on Fire of the False + Pants on Fire, then compared that percentage between Democrats and Republicans as a way to show how many more Pants on Fire are given to Republicans. It appears that PolitiFact Bias’ Bryan White may have gotten the idea from Eric Ostermeier’s much lauded study where he also took a look at a similar percentage (and probably Uppity Wisconsin did as well, in terms of True and Mostly True).

Overall results pointed to a marked liberal bias from PolitiFact’s start in 2007 through the end of 2011. The percentages for groups A and B varied only slightly. For group A, Republicans found making false statements received a “Pants on Fire” rating 28.57% of the time and Democrats 16.42 . For group B the percentages were 28.23% and 15.88%, respectively--a very close match. Group C reflected the expected distortion from unreliable chain email claims. Though neutral judgment ought to result in approximately equal proportions of unfair “Pants on Fire” ratings, Republicans drew that designation about 74% more often than Democrats. The numbers show a clear bias against Republicans and conservatives.

For further study and comparison I will use the percentage of one party’s “Pants on Fire” ratings divided by the percentage of the other party’s “Pants on Fire” ratings as the “PoF Bias” number. For Group A from 2007 through the end of 2011 Republicans suffered with a 1.74 PoF Bias number.

I’m not sure how he could place such confidence or reliance in such a number as being “a clear bias against Republicans” or how in the world could he expect that “neutral judgment ought to result in approximately equal proportions of unfair Pants on Fire ratings.”

To repeat PolitiFact Editor Bill Adair’s own words from the PolitiFact New Jersey fact-check:

PolitiFact Editor Bill Adair called suggestions that a governor is "in the top five" for falsehoods "inaccurate -- and meaningless.” ….

"It's fine if people want to use our report cards to compare the records of people we fact-check. But it's important to be accurate about what those report cards say and what they don't," Adair said. "The report cards provide a tally of the claims we chose to check. But it's not accurate to say the report cards indicate who ‘told the most lies.’ We are journalists who choose our fact-checks based on what is newsworthy. We are not social scientists and are not using any kind of random sample to select statements to check."

Adair re-iterated his “selection bias”—what is newsworthy—but according to him, tallying up the rulings to make comparisons of truthfulness is “inaccurate – and meaningless.” I would venture Adair would say the same thing even with tallying up claims at a macro level as done by White.

While I’ve taken a lot of guesses—it could be liberal bias, it could be Republicans lie more, it could be the rise of Fox News or the Tea Party, it could be a lot of things—but I don’t think just adding up the Pants on Fire as a percent of all claims found false validates any conclusion of liberal bias.   There's no  PolitiFact "Achilles' Heel" here as White wants his readers to believe.

So, since I had the same database, I decided to see if I could first come close to confirming White’s results….and I did, only my number was a bit higher at 1.79 (his was 1.74). This was for all PolitiFact National claims through the end of 2011. A lot of minor things could have caused the variance, in particular the classification of political affiliation for chain e-mails, many of which were False and Pants on Fire. White also delineated conservatives from Republicans, which could be a very subjective process. I never separated conservatives from Republicans just as I never separated liberals from Democrats. Nevertheless, it was very close.

But the PolitiFact (PF) states, one would surmise, should be pretty close to that of National; after all, they were likely “trained” in the process of fact-check by PF National, they rate national figures (Eric Cantor by PF Virginia and John Boehner by PF Ohio, for example), and PF Florida is basically a clone of PF National (read: Angie Drobnic Holan). They exceed PF National in number, comprising 53% of all rulings. Plus, we have the office holder distinction as used by Eric Ostermeier: it “adjusts” out the “expected distortion of chain email claims” as well as media pundits, advocacy groups, PACs and websites.

Next, I thought I should include the Truth Index for the identical data which created the “PoF bias” factor, which takes into account the other Truth-o-Meter categories, to see if there was any correlation. After all, if there’s more Falses and Pants on Fire, it should follow there are fewer Trues and Mostly Trues, since it’s basically a zero sum game. I did two forms of three scenarios, marked A, B and C.

The “A” scenario was where I rated all Falses and Pants on Fire for PolitiFact National to get to White’s “liberal bias” factor. I then computed the number for “Office Holders” which reduced the “PoF bias” factor to 1.55. The Truth Index showed a roughly 30 point variance in favor of the Democrats for both categories.

The “B” scenario was for PolitiFact National plus all the State PolitiFacts. There were 111% more rulings because, as noted, the PolitiFact states produce more than half of PolitiFact’s fact-checks. Overall, the “liberal bias factor” was now at 1.21; for office holders, it was 1.05. That means, if this was White’s calculations at .05 less, or about .025 adjusting for the “all” PolitiFact count, we may now be very close to the “perfect “ 1.00 for “neutrality” for all PolitiFact office holders. However, the Democrats are still favored in the Truth Index by a variance of 23 points in both categories.

The “C” scenario is for PolitiFact states only, and excludes PolitiFact National. Again, this would be for more rulings than in the “A” Scenario. At this point the “PoF Bias” factor is .96, or reverses to a conservative “PoF Bias” factor of 1.04 depending on how you want to look at it. But the Truth Index still favors Democrats by a margin of 14 points. For office holders, the liberal “PoF Bias” factor gets cut even more to .83 (or a conservative “PoF bias” factor of 1.17) which is no longer close to “neutral”. But the Truth Index still favors the Democrats, by an even larger margin of 17 points.

c

Click to enlarge:  Red shading shows "PoF bias" factor favors Repubs, blue the Dems; note Truth Index at bottom
still favors Dems even though "PoF bias" factor favors Republicans

Conclusions? White claims that “When one party ends up with a PoF Bias number greater than 1.0 the number indicates a bias against that party in the application of the ‘Pants on Fire’ rating.” Maybe, with what appears to be a close to 1.00 “neutral or no bias” overall, it appears that the PolitiFact regions assist National to (as White says) compensate for the Falses and Pants on Fire in order to “manage audience perceptions.” Maybe...if such a score really shows bias.  But that still doesn’t answer the question of the Truth Index, which regardless of the “PoF Bias” factor, seems to always weigh to the Democrats’ favor. This means that the Democrats have more Trues and Mostly Trues, enough more than the Republicans, to outweigh the effect of the Falses and Pants on Fires.

I would also advise White not to read too much into his theory that the early 2007 rulings, from when PolitiFact was working with the Congressional Quarterly, would have scored as less biased at that time. I’ve found scores favoring the Republicans in a variety of scenarios as I’ve worked the rulings data, and none of them have been very consistent, which tells me that there is no purposefulness to their selection of claims or their assignment of scores.

Taking a look at the Pants on Fire rulings for 2007 (from August through December), there were only ten, six of them written by Angie Drobnic Holan and one by Bill Adair. Ten rulings are like ten rulings on an individual: there's too little to go by, so they tell you nothing. But if seven out of ten of those rulings were done by those who are managing PolitiFact today, I don’t know how White can claim PolitiFact was more neutral back in its beginnings.

All he has found is that when PolitiFact National makes a False ruling on a Republican, they are 74% more likely to make the False Pants on Fire, than they are for a Democrat. This doesn’t mean they’re doing it because they are liberally biased. To prove it’s due to that is a whole new ball game. He basically has to prove that the whole PolitiFact National crew are dyed-in-the-wool liberals who have an agenda, which isn't even remotedly what he did here.  As Eric Levine notes in his blog “Content in Reality”: “It should be pretty clear that merely tallying Politifact ratings doesn't give you anything remotely conclusive about the theoretical bias of a site…if you are having THAT much trouble exposing bias, maybe you should wonder if the bias really exists at all...”