I Brake for Pants on Fire (in 2013)

Since its debut in 2007 through the end of 2012, the PolitiFact team (including its states) decided that 1,915 statements it fact-checked out of the 6,658 total it had done deserved a false.  And of these statements found false, 29.6 percent, or 566, were deserving of its now famous Pants on Fire designation.  While I'm not going to discuss the somewhat subjective exercise of how it's assigned, it may be worth unpacking it and seeing what's inside.

 

If those 566 Pants on Fire rulings, 137 or 24% went to Democrats, 369 or 65% went to Republicans; the rest (60) were non-partisan or unknown as to a party affiliation.  This is what "alarms" to some degree the PolitiFact detractors who are mostly Republican: on a "raw" ruling basis, Republicans (in general) were 2.71 times more likely to be assigned the Pants on Fire than the Democrats.    The Republican theories as to the reason for this have been sampling or selection bias, although there are other compelling ones which I've posted about before:  the loss of the Whitehouse in 2008, and the enacting of "Obamacare."

 

One particular categorizing that was first done by the University of Minnesota's Eric Ostermeier in a study of 511 PolitiFact rulings in 2010 was "official versus non-official"; it took into account those who were in office.  Only 190, or about one third, of the Pants on Fire rulings were awarded to people who were in office; of the other 376 who were not in office, half were organizations and the on-line media.  If we look at party affiliation of those in office who were assigned the Pants on Fire, we have 60 Democrats (32 percent) and 128 Republicans (67 percent).  This means Republicans in office were 2.1 times more likely to be awarded the Pants on Fire than were Democrats in office, still more than double but about 25 percent less than the 2.71 overall.

 

The media--which includes celebrities, pundits, blogs, websites and chain e-mails--received the largest percentage (24%) of Pants on Fire claims, followed by governors and state government officials (19%).

 

Sixty-eight rulings receiving the Pants on Fire related to Obamacare, the vast majority from Republicans.  Claims on the subject of healthcare generated the most Pants on Fire rulings, with over 80 percent of them about Obamacare, or 14 percent of the Pants on Fire claims.   Obamacare overlapped to the subjects of Medicare, housing and abortion as well.  As a general topic, however, fiscal subjects (deficit, economy, federal/state budgets, stimulus and taxes) took up about 20 percent of the Pants on Fire claims.

 

As the number of fact-checks has grown year to year, however, the percentage of Pants on Fire claims in terms of the total has been gradually dropping year to year since "peaking" in 2009 as shown here.

Click to enlarge:  Number of rulings shown in parentheses after year

Finally, below are the same numbers which show the Republican/Democrat breakdown each year. What this appears to show is that while the percentage of Pants on Fire overall has gone down, the percentage of those Pants on Fire assigned to Republicans has gone up.  Non-official candidates--which include Mitt Romney and the other Republican presidential candidates like Herman Cain and Rick Santorum--comprise the third highest category in terms of candidate occupation for Pants on Fire rulings, which may have contributed to this high percentage in 2011 and 2012.  So, we may see a drop in this high Republican percentage in 2013.  Stay tuned.

Click to enlarge: Years (2010-12) of most Pants on Fire were dominated by Republicans

 

Quantifying the PolitiFact Quants

There's certainly a lot of people out there scoring PolitiFact rulings... I'm finding more and more of them every day.  Presented and scored in different ways, in studies, articles and blogs, I think it may be time to add them as a special section to my PolitiFact Detractors page, though "detractors" are not quite what they are.

While I hope I'm one of the better ones, I do need to make the case.  I have probably looked at these rulings in the largest variety of ways, and have done so on an ongoing basis for quite a while (since 2010, when I started this blog).  I've looked at the score in terms of writer, topic, local versus national, by individual compared with other fact-checkers, etc.   I'm looking for patterns.  For example, if Republicans lie more, do they lie about every subject?  (No, there are some subjects for which they are more truthful than Democrats)  Are lies more ideologically based? (Yes)  Do some PolitiFact writers appear to favor one party over the other based on the Truth Index? (Yes) Most of the other PolitiFact quants have looked at them in terms of just the presidential contenders or Democrat versus Republican overall.  Some qualify their rulings as far as sampling/selection bias while others take the rulings at face value.

Straight Politi-Quant websites.  The website Who's More Full of Sh*t (authored by J. Kalgreen) graphically rates the same....with an ongoing feed from PolitiFact.  It uses a score heavily weighted to the negative rulings.  It also gives an overall Democrat versus Republican rating.  While it's a pure rating website, its website set up is quite sophisticated.  As noted by Benjamin "Brash Equilibrium" Chabot-Hanowell, "I seriously wish I could code CSS and PHP as well as J. Kalgreen [WMFoS author] can."

While a great site, Who's More Full of Sh*t might not be linked by PolitiFact because its name might offend some.

Instead of saying who's more "truthier" or "honest" it goes in the opposite direction--scoring zero for True down to 200% for Pants on Fire.  So if my conservative counterpart decided to try to troll-comment the site (which he can't because no comments are enabled) he'd have to complain that PolitiFact's selection bias is the cause for Democrats being less full of sh*t, or for Republicans to be more full of sh*t.  Because, you see, both are still full of sh*t.

Malark-o-Meter is more of the nerdy Nate Silver variety of  meta/probability analysis and in that respect is like no other PolitiFact quantifier.  A newer website based on (I'm guessing) Joe Biden's outburst at the vice presidential debate, author Benjamin "Brash Equilibrium" Chabot-Hanowell is interested in mathematically determining a possible "bias" factor as well as the probability of how much each claimant is prone to lie. He includes the Washington Post Factchecker rulings in his analysis as well.

"Serious" Studies.  I've posted about and often referred often to Eric Ostermeier's 2010 study of 511 PolitiFact rulings  where he concluded with the infamous selection bias statement.

The Center for Media and Public Affairs did a study of both PolitiFact and Washington Post Factchecker rulings but did not really reach any other conclusion ("[took] no position on their accuracy") as to why Republicans score as less truthful than Democrats.

And of course, the website PolitiFact Bias did a research paper on PolitiFacts False/Pants on Fire rulings, coming up with a percentage factor that I often use in my blog.    There are others who have looked at the Pants on Fire, some in a humorous way, or a lot like PolitiFact Bias.

Website Articles.  One of the very first to quantify PolitiFact's Truth-o-Meter rulings was the pseudonymous "Cinematical" at the Daily Kos, back in 2009....and arrive at the conclusion as so many that would follow, that conservatives were less "honest" than liberals.  There's also been several scorings at the Daily Beast, as well as a nice quant review by the New York Times' FiveThirtyEight.com (Nate Silver's blog).

Blogs.  Anyone math-oriented knew the moment they saw the Truth-o-Meter that it begged to be measured.  I imagine there's a lot more than just those mentioned here. So (besides myself), here's three very good ones:

In 2011, I came across the blog Quibbling Potatoes authored by Steve, who had some great ideas for categorizations of the rulings that I now use for my data.  He also had a name for his score which he called...the Truth Index.  While I don't think PolitiFact took the name from Steve's site, one can certainly say Steve had a bit of clairvoyance.  It's obvious he put a lot of work into his charts and graphs and it would be nice to see him do it again.

Pandering Politicians is a blog (turned website like PolitiFact Bias) by someone who calls himself "PanderBear" and the Truth Index here is called the "Truth Quotient."  He has a lot of charts, especially time trendlines, and begs people not to even read chain e-mails because of their dismal "TQ."  And of course, in one of the comment sections we see the ubiquitous question from my conservative counterpart asking what he does about selection bias.  No answer from PanderBear.  At least as far as I could see.

Allen Allermann in a namesake blog entitled "Random Human Neural Firings" did a quantifying review about the same time as Steve called "Who do You Trust?"  He scored PolitiFact rulings in the opposite direction that many tend to do--he gave higher weight to the True and Mostly True instead of False and Pants on Fire.  He uses a "sports" analogy for what Brash Equilibrium terms the report cards of those rated by PolitiFact; he calls them "at bats."

So there we have, including my own, about a dozen attempts to quantify PolitiFact rulings.  Perhaps two qualifiers can be added: (1) None of these websites/studies/blogs are affiliated with or approved by PolitiFact (as far as I know) and (2) all numbers should be interpreted with caution.  And if you come across any websites/articles/blogs that quantify PolitiFact's Truth-o-Meter rulings, please let me know in the comment section.

PolitiFact Detractors: WatchDog? More Like Watch-Mutt!

The latest in PolitiFact detracting, in this case  PolitiFact Ohio detracting, comes from something called Watchdog.Org which is part of “The Franklin Center for Government and Public Integrity.” Of course, my conservative counterpart starts out his post with the Franklin Center’s honorable-sounding name as the sentence subject to give his post some false stature even though the Watchdog.Org website includes this disclaimer:

The Franklin Center is not responsible for the information that appears on the watchdog sites. The organization serves as a capacity builder and networking agent for independent, state-based journalists and organizations…”

Most of the PolitiFact-is-opinion-parading-as-fact series they’ve done, by two “journalists” Jon Cassidy and Will Swaim, consist of some very narrow interpretations of the ruling along with a lot of condescension toward PolitiFact….about adding numbers, understanding economics, finance, etc., as if the authors were some sort of experts themselves, for which there was pretty scant evidence. Usually only one link, to the PolitiFact ruling, was included in their “article.”

PolitiFact Bias, however, takes no notice of the excessive snark (their own name for it) and does their usual cheer leading routine: “Jon Cassidy blasts PolitiFact Ohio….” in one and “absolutely clobbers” them in another. I guess snark is okay by writers for the “Franklin Center for Government and Public Integrity” when talking about PolitiFact, but a slight implication of it from PolitiFact is a seering indictment of bias.

In some of them, Swaim or Cassidy argue the difference between True and Mostly True. Now, to find something categorically false, or to argue the degree of falsity, is one thing, but the True to Mostly True or vice versa is sort of like arguing which horse is the best horse in the glue factory. I’m only going to cover a few of their criticisms here, but will include links to all of them. I liken some of the writing to my own: I consider myself an amateur at “journalism” because I have no education or training in it. In other words, Cassidy and Swaim aren’t very professional….well maybe professional hacks, you never know. They all start with the identical copy and paste intro:

PolitiFact Ohio practices opinion journalism under the guise of fact-checking. They often get things wrong — particularly, we’ve noticed, in their coverage of U.S. senate candidates Sherrod Brown (Democrat) and Josh Mandel (Republican). So we bring you PolitiFact or Fiction, a semi-regular review of pronouncements issued by PolitiFact Ohio, a blog run by staff at the Cleveland Plain Dealer and supported by Politifact.com.

They hit the ground running with their penchant for snark in the very first critique by Jon Cassidy, comparing two statements related to the same subject by Josh Mandel, a Republican candidate (State of Ohio Treasurer) running against Democrat Ohio Senator Sherrod Brown, with “Does anyone at Cleveland Plain Dealer understand finance?”   It's in the flimsyTrue-or-Mostly-True venue first mentioned.

Truth Index: Smart Politics Revisited

Early last year I did a pretty detailed analysis on Eric Ostermeier’s study of about a years’ worth of PolitiFact (PF) National rulings. I compared my larger numbers to his at that time; to be specific, Ostermeier evaluated 511 PolitiFact statements while my sample was 1,784. My number actually included everything that was in his 511.

This time I am again looking at the entire population of PolitiFact rulings to compare to Eric Ostermeier’s. The most notable and most cited by various conservative pundits in terms of its possible bias is the 76/22 one: that is, 76% of all False and Pants on Fire rulings are attributable to Republicans while only 22% are attributable to Democrats. Mark Hemingway of the conservative Weekly Standard cited it in a recent NPR radio piece with Washington Post Fact-Checker author Glenn Kessler.  Here's Eric's finding:

Republican statements were graded in the dreaded "false" and "pants on fire" categories 39 percent of the time, compared to just 12 percent for statements made by Democrats.

That means a supermajority of falsehoods documented by PolitiFact over the last year - 76 percent - were attributed to Republicans, with just 22 percent of such statements coming from Democrats.

So, how does it look now? I could never get close to Ostermeier’s numbers, although Republican False and Pants on Fire always outnumbered the Democrats. I did it in several ways to compare to Ostermeier: one with everything, which would include the PF states, one with everything including the non-office holders, and one just on National. I actually achieved the 22% for the Democrats by using the top 60 in terms of those with 10 or more rulings as posted previously on individual Truth Index average; but that is not a representative sample. The Republicans in this case were just over 70%, not 76%. But in all the other more inclusive measures, as shown in the graph below, the Republicans varied from around 60% (all False and Pants on Fire rulings) to 69% (PF National only). The Democrats were anywhere from 26% to almost 30% for all False and Pants on Fire rulings. Here’s a graphic of all my “permutations,” including one which included everyone and everything: for the 4,707 rulings PF did from the day it was “born” through the end of 2011, the False and Pants on Fire categories were roughly 60% attributable to Republicans, 30% to Democrats, with the remaining 10% for Independents, non-partisans and unknowns. While it’s still a 2 to 1 ratio in favor of the Democrats, it’s clearly better than the 3.5 to 1 as found by Ostermeier.

Click to enlarge:  Smart Politics' percentages for Republican "Falsehoods" exceeded all five ways I did it with the entire PF database. (NH is New Hampshire, which appears to only do national rulings related to the 2012 Elections.)

The other Truth-o-Meter categories were reviewed by Ostermeier as well, and so I obliged with my own, calculating not only just the percent of total rulings compared, but the rulings as a percent of all those in the Truth-o-meter rating. Again, this was for elected officials and those holding office. According to Ostermeier, "....… Democrats have therefore been presented as much more truthful - with over 75 percent of statements receiving the top three grades of True (16 percent), Mostly True (27 percent), or Half True (33 percent)."

The numbers indicate that Ostermeier’s sample appeared to be skewed more in favor of Democrats, no matter how I tried to shake the numbers. I had 10 fewer points for the Democrats and 5 more for the Republicans, and there was even one stat in favor of the Republicans: that was the percent True, Mostly True and Half True as a percent of the total True, Mostly True, and Half True: Of the 2,033 rulings in those Truth-o-Meter categories, 50% were attributable to Republicans, while 47% were attributable to Democrats. Part of the reason may be, of course, that there were more Republican rulings. While at the time Ostermeier found that PF National had roughly the same number of rulings for each party in his sample, they began to become more heavily Republican with the 2010 Elections putting more Republicans in office, and the 2011 Republican presidential campaigns and debates requiring frequent fact-checks. As a result, Republican rulings increased about 10 percent while Democrat rulings dropped almost 14 percent from Ostermeier’s more even percentages.

Next, and last but not least, a few fact-ettes and fact-oids shared from my database of PolitiFact rulings.

Sidebar: Preconception Detection

It’s been proven in my blog to some degree that any sort of bias by PolitiFact (PF) in its rulings can’t be proven through “a preponderance of anecdotal evidence.” Because (1) some of that preponderance is clearly arguable, as I have shown in Lil White Lies, and (2) the evidence of bias can go both ways, as I have shown in Grading PF Liberal Style, as well as in my Guide to PF Detractors, which lists those alleging PF of bias on both sides of the aisle.

In my many searches for items to add to my Guide to PF Detractors, I occasionally look at comments, to see if someone has anything original to say (generally it’s run of the mill). Here were two, however, that really offer some different approaches to determine bias by PolitiFact.

They come from an article by David Brauer at MinnPost.com who was critiquing Bill Adair’s response to Eric Ostermeier’s study in which he suggests that PolitiFact writers may be using “selection bias” in choosing statements for review. (I reviewed Ostermeier’s numbers, comparing them to mine, and found that his Republican “negatives” were somewhat higher in his database of 512 rulings as compared to my database of 1,330 rulings on current and former office-holders.)

Here’s the first one written by Mark Gisleson: (emphasis added)

Politifact isn't biased by party, they're biased in an establishment way. That's why Bachmann does badly by them, but so do people like Alan Grayson who they often parsed to death.

They also cannot differentiate between politicians who skillfully dissemble, as opposed to politicians who equivocate themselves into gray areas.

They would do much better if they placed every politician's statements in the context of their party's political agenda. If a misstatement serves their agenda, it should be graded more harshly because it's fair to assume that lies that benefit the cause are deliberate. 

I’ve already noticed that my conservative counterpart might be aware of this, because, in his Grading PolitiFact critiques, he often seems to deliberately and assiduously avoid placing the argument in the context of the (Republican) party’s political agenda. An excellent case in point is a PF ruling on Ann Coulter reviewed earlier this year where, following the tsunami-nuclear reactor incident in Japan, she claimed to Bill O’Reilly that radiation was good for you. Bryan’s argument that she didn’t deserve a “False” from PF was that the context of Coulter’s argument was the media’s over-reaction to it, when the fact was, as I showed, they seemed to be under-reacting, if anything, and the real context was the Republican party’s promotion of nuclear energy, ergo, Coulter’s claims about radiation. Bryan didn’t say anything about the Republican advocacy of nuclear energy.

Politi-Score: "My Bad" Boehner

In my initial review of Eric Ostermeier’s article “Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats” I deduced that he had used PolitiFact National statistics only. As a matter of uniformity, this was probably the most “scientific” way to do it. But because his total number of 511 statements still seemed a little low, and he did not come out and say that this was PolitiFact National, I puzzled over how I could determine it.

Since Speaker of the House John Boehner is from Ohio, I thought I might manually confirm his rulings. Indeed, he had some rulings by PolitiFact National and some by PolitiFact Ohio, here’s how his count worked out for 2010 when he received 21 PolitiFact rulings:

Boehner 2010	PF National	PF Ohio	Total
True	6	4	10
Mostly True	2	0	2
Half True	1	2	3
Barely True	1	0	1
False	3	2	5
Pants on Fire	0	0	0
Total	13	8	21

So right there (in emphasis), as Ostermeier noted, six Trues, two Mostly Trues, and one Half True. Yes, Ostermeier had compiled Boehner’s numbers for PolitiFact National only.

The issue raised for me was the way I had gathered my data for my “2,788 Reasons” series of posts. I accomplished the task by going to the “Other People and Groups” page of the PolitiFact.com website, and linking to each person, and then copying the numbers from the “Statements by Ruling” list. But the issue was that the “Statements by Ruling” do not tell you which PolitiFact group the ruling was done by, and there are some overlaps. The other politico I checked was House Majority Leader Eric Cantor, since he is a representative from Virginia, and yes, he had one ruling done by PolitiFact Virginia.

Politi-Score: Dumb Politics in Detail

I wasn’t quite satisfied with the analysis of my previous post on Smart Politics’ Eric Ostermeier’s critique on PolitiFact rulings. I decided to do something a little more in depth and graphical, and to look into my contention that he only used PolitiFact National’s ruling data. Some of this post repeats some of the findings, but puts them in a more understandable format. And, in the process, I found out I needed to make some adjustments to the database I had collected because of an interesting finding.

Of the 511 statements put through the Truth-O-Meter test from January 1, 2010 through January 31, 2011, PolitiFact devoted 74 percent of its attention to current and former political officeholders and elected officials (379 statements), 17 percent to ideological organizations and individuals not holding political office (85 statements), and 9 percent to other groups and individuals without a partisan or ideological agenda (28 statements). Another 20 statements came from chain e-mails, public opinion polls, bumper stickers, or "bloggers" generally (4 percent).

Source of PolitiFact Ruling (PolitiFact National)	Ostermeier Count	Ostermeier Percentage	Politi- Psychotics Count	Politi- Psychotics Percentage
Current and former political officeholders and elected politicians	379	74.2	1,330	74.6
Ideological organizations and individuals not holding political office	85	16.6	316	17.7
Other groups and individuals without a partisan or ideological agenda	28	5.5	34	1.9
Chain e-mails, public opinion polls, bumper stickers or “bloggers”	20	3.9	104	5.8

His numbers add to 512 (one more than 511), and the chain e-mails, etc., overlap with groups and individuals without a partisan or ideological agenda. In other words, when he said “9 percent to other groups and individuals” it includes the 20 statements from chain e-mails, etc., since his quoted 28 does not compute to 9 percent of 511. I’ve divided that 9 percent to show he had about 5 percent for those without an ideological agenda, and about 4 percent for chain e-mails, etc.

Politi-Score: Dumb Politics

Recently there’s been some hubbub now that more notable forums are trying their hand at crunching PolitiFact’s rulings, and arriving at similar conclusions, with a piece by one Eric Ostermeier at a website called “Smart Politics” which is part of the University of Minnesota website. (U of M???? I prefer the Wolverine-type, actually.)

The title of Ostermeier’s article is, “Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats.” Looking over his tables, which are presented in reverse with Pants on Fire first, I was immediately struck by the fact that the 511 total number wasn’t enough--even if he was just counting “Current and Former Political Officials.”

A Smart Politics content analysis of more than 500 PolitiFact stories from January 2010 through January 2011 finds that current and former Republican officeholders have been assigned substantially harsher grades by the news organization than their Democratic counterparts.

Ostermeier supposedly tallied these 511 “statements put through the Truth-o-Meter test from January 1, 2010 through January 31, 2011.” That’s 13 months. Therefore, my educated guess is that he only counted the rulings of PolitiFact National.

Ostermeier segregates the rulings similarly to the method I have followed: the “current and former political officials” from those of “pundits, groups, emails, blogs and miscellaneous” rulings.  It appears that he wanted to determine if the pundits were graded more “harshly” given they would not be driven by voter appeal. But his conclusions are clear on the “current and former political officials”:

Republican statements were graded in the dreaded "false" and "pants on fire" categories 39 percent of the time, compared to just 12 percent for statements made by Democrats.

That means a supermajority of falsehoods documented by PolitiFact over the last year - 76 percent - were attributed to Republicans, with just 22 percent of such statements coming from Democrats.

As a consequence, Democrats have therefore been presented as much more truthful - with over 75 percent of statements receiving the top three grades of True (16 percent), Mostly True (27 percent), or Half True (33 percent).

Less than half of Republican statements graded by PolitiFact were regarded as half truths or better - just 90 out of 191 (47 percent).

Now that he’s given me his breakdown, I can compare it with mine.  First my data needed to be filtered to PolitiFact National.  Secondly,  the pundits, orgs, emails, etc., were separated out so my measure would match his criteria.   The comparison reflects two different periods of time:  Ostermeier's is for the 2010 calendar year, while mine is for PolitiFact rulings since its inception in 2007 through the end of 2010.