Truth Index: Smart Politics Revisited

Early last year I did a pretty detailed analysis on Eric Ostermeier’s study of about a years’ worth of PolitiFact (PF) National rulings. I compared my larger numbers to his at that time; to be specific, Ostermeier evaluated 511 PolitiFact statements while my sample was 1,784. My number actually included everything that was in his 511.

This time I am again looking at the entire population of PolitiFact rulings to compare to Eric Ostermeier’s. The most notable and most cited by various conservative pundits in terms of its possible bias is the 76/22 one: that is, 76% of all False and Pants on Fire rulings are attributable to Republicans while only 22% are attributable to Democrats. Mark Hemingway of the conservative Weekly Standard cited it in a recent NPR radio piece with Washington Post Fact-Checker author Glenn Kessler.  Here's Eric's finding:

Republican statements were graded in the dreaded "false" and "pants on fire" categories 39 percent of the time, compared to just 12 percent for statements made by Democrats.

That means a supermajority of falsehoods documented by PolitiFact over the last year - 76 percent - were attributed to Republicans, with just 22 percent of such statements coming from Democrats.

So, how does it look now? I could never get close to Ostermeier’s numbers, although Republican False and Pants on Fire always outnumbered the Democrats. I did it in several ways to compare to Ostermeier: one with everything, which would include the PF states, one with everything including the non-office holders, and one just on National. I actually achieved the 22% for the Democrats by using the top 60 in terms of those with 10 or more rulings as posted previously on individual Truth Index average; but that is not a representative sample. The Republicans in this case were just over 70%, not 76%. But in all the other more inclusive measures, as shown in the graph below, the Republicans varied from around 60% (all False and Pants on Fire rulings) to 69% (PF National only). The Democrats were anywhere from 26% to almost 30% for all False and Pants on Fire rulings. Here’s a graphic of all my “permutations,” including one which included everyone and everything: for the 4,707 rulings PF did from the day it was “born” through the end of 2011, the False and Pants on Fire categories were roughly 60% attributable to Republicans, 30% to Democrats, with the remaining 10% for Independents, non-partisans and unknowns. While it’s still a 2 to 1 ratio in favor of the Democrats, it’s clearly better than the 3.5 to 1 as found by Ostermeier.

Click to enlarge:  Smart Politics' percentages for Republican "Falsehoods" exceeded all five ways I did it with the entire PF database. (NH is New Hampshire, which appears to only do national rulings related to the 2012 Elections.)

The other Truth-o-Meter categories were reviewed by Ostermeier as well, and so I obliged with my own, calculating not only just the percent of total rulings compared, but the rulings as a percent of all those in the Truth-o-meter rating. Again, this was for elected officials and those holding office. According to Ostermeier, "....… Democrats have therefore been presented as much more truthful - with over 75 percent of statements receiving the top three grades of True (16 percent), Mostly True (27 percent), or Half True (33 percent)."

The numbers indicate that Ostermeier’s sample appeared to be skewed more in favor of Democrats, no matter how I tried to shake the numbers. I had 10 fewer points for the Democrats and 5 more for the Republicans, and there was even one stat in favor of the Republicans: that was the percent True, Mostly True and Half True as a percent of the total True, Mostly True, and Half True: Of the 2,033 rulings in those Truth-o-Meter categories, 50% were attributable to Republicans, while 47% were attributable to Democrats. Part of the reason may be, of course, that there were more Republican rulings. While at the time Ostermeier found that PF National had roughly the same number of rulings for each party in his sample, they began to become more heavily Republican with the 2010 Elections putting more Republicans in office, and the 2011 Republican presidential campaigns and debates requiring frequent fact-checks. As a result, Republican rulings increased about 10 percent while Democrat rulings dropped almost 14 percent from Ostermeier’s more even percentages.

Next, and last but not least, a few fact-ettes and fact-oids shared from my database of PolitiFact rulings.

Politi-Score: "My Bad" Boehner

In my initial review of Eric Ostermeier’s article “Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats” I deduced that he had used PolitiFact National statistics only. As a matter of uniformity, this was probably the most “scientific” way to do it. But because his total number of 511 statements still seemed a little low, and he did not come out and say that this was PolitiFact National, I puzzled over how I could determine it.

Since Speaker of the House John Boehner is from Ohio, I thought I might manually confirm his rulings. Indeed, he had some rulings by PolitiFact National and some by PolitiFact Ohio, here’s how his count worked out for 2010 when he received 21 PolitiFact rulings:

Boehner 2010	PF National	PF Ohio	Total
True	6	4	10
Mostly True	2	0	2
Half True	1	2	3
Barely True	1	0	1
False	3	2	5
Pants on Fire	0	0	0
Total	13	8	21

So right there (in emphasis), as Ostermeier noted, six Trues, two Mostly Trues, and one Half True. Yes, Ostermeier had compiled Boehner’s numbers for PolitiFact National only.

The issue raised for me was the way I had gathered my data for my “2,788 Reasons” series of posts. I accomplished the task by going to the “Other People and Groups” page of the PolitiFact.com website, and linking to each person, and then copying the numbers from the “Statements by Ruling” list. But the issue was that the “Statements by Ruling” do not tell you which PolitiFact group the ruling was done by, and there are some overlaps. The other politico I checked was House Majority Leader Eric Cantor, since he is a representative from Virginia, and yes, he had one ruling done by PolitiFact Virginia.

Politi-Score: Dumb Politics in Detail

I wasn’t quite satisfied with the analysis of my previous post on Smart Politics’ Eric Ostermeier’s critique on PolitiFact rulings. I decided to do something a little more in depth and graphical, and to look into my contention that he only used PolitiFact National’s ruling data. Some of this post repeats some of the findings, but puts them in a more understandable format. And, in the process, I found out I needed to make some adjustments to the database I had collected because of an interesting finding.

Of the 511 statements put through the Truth-O-Meter test from January 1, 2010 through January 31, 2011, PolitiFact devoted 74 percent of its attention to current and former political officeholders and elected officials (379 statements), 17 percent to ideological organizations and individuals not holding political office (85 statements), and 9 percent to other groups and individuals without a partisan or ideological agenda (28 statements). Another 20 statements came from chain e-mails, public opinion polls, bumper stickers, or "bloggers" generally (4 percent).

Source of PolitiFact Ruling (PolitiFact National)	Ostermeier Count	Ostermeier Percentage	Politi- Psychotics Count	Politi- Psychotics Percentage
Current and former political officeholders and elected politicians	379	74.2	1,330	74.6
Ideological organizations and individuals not holding political office	85	16.6	316	17.7
Other groups and individuals without a partisan or ideological agenda	28	5.5	34	1.9
Chain e-mails, public opinion polls, bumper stickers or “bloggers”	20	3.9	104	5.8

His numbers add to 512 (one more than 511), and the chain e-mails, etc., overlap with groups and individuals without a partisan or ideological agenda. In other words, when he said “9 percent to other groups and individuals” it includes the 20 statements from chain e-mails, etc., since his quoted 28 does not compute to 9 percent of 511. I’ve divided that 9 percent to show he had about 5 percent for those without an ideological agenda, and about 4 percent for chain e-mails, etc.

Politi-Score: Dumb Politics

Recently there’s been some hubbub now that more notable forums are trying their hand at crunching PolitiFact’s rulings, and arriving at similar conclusions, with a piece by one Eric Ostermeier at a website called “Smart Politics” which is part of the University of Minnesota website. (U of M???? I prefer the Wolverine-type, actually.)

The title of Ostermeier’s article is, “Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats.” Looking over his tables, which are presented in reverse with Pants on Fire first, I was immediately struck by the fact that the 511 total number wasn’t enough--even if he was just counting “Current and Former Political Officials.”

A Smart Politics content analysis of more than 500 PolitiFact stories from January 2010 through January 2011 finds that current and former Republican officeholders have been assigned substantially harsher grades by the news organization than their Democratic counterparts.

Ostermeier supposedly tallied these 511 “statements put through the Truth-o-Meter test from January 1, 2010 through January 31, 2011.” That’s 13 months. Therefore, my educated guess is that he only counted the rulings of PolitiFact National.

Ostermeier segregates the rulings similarly to the method I have followed: the “current and former political officials” from those of “pundits, groups, emails, blogs and miscellaneous” rulings.  It appears that he wanted to determine if the pundits were graded more “harshly” given they would not be driven by voter appeal. But his conclusions are clear on the “current and former political officials”:

Republican statements were graded in the dreaded "false" and "pants on fire" categories 39 percent of the time, compared to just 12 percent for statements made by Democrats.

That means a supermajority of falsehoods documented by PolitiFact over the last year - 76 percent - were attributed to Republicans, with just 22 percent of such statements coming from Democrats.

As a consequence, Democrats have therefore been presented as much more truthful - with over 75 percent of statements receiving the top three grades of True (16 percent), Mostly True (27 percent), or Half True (33 percent).

Less than half of Republican statements graded by PolitiFact were regarded as half truths or better - just 90 out of 191 (47 percent).

Now that he’s given me his breakdown, I can compare it with mine.  First my data needed to be filtered to PolitiFact National.  Secondly,  the pundits, orgs, emails, etc., were separated out so my measure would match his criteria.   The comparison reflects two different periods of time:  Ostermeier's is for the 2010 calendar year, while mine is for PolitiFact rulings since its inception in 2007 through the end of 2010.