Quantifying the PolitiFact Quants

There's certainly a lot of people out there scoring PolitiFact rulings... I'm finding more and more of them every day.  Presented and scored in different ways, in studies, articles and blogs, I think it may be time to add them as a special section to my PolitiFact Detractors page, though "detractors" are not quite what they are.

While I hope I'm one of the better ones, I do need to make the case.  I have probably looked at these rulings in the largest variety of ways, and have done so on an ongoing basis for quite a while (since 2010, when I started this blog).  I've looked at the score in terms of writer, topic, local versus national, by individual compared with other fact-checkers, etc.   I'm looking for patterns.  For example, if Republicans lie more, do they lie about every subject?  (No, there are some subjects for which they are more truthful than Democrats)  Are lies more ideologically based? (Yes)  Do some PolitiFact writers appear to favor one party over the other based on the Truth Index? (Yes) Most of the other PolitiFact quants have looked at them in terms of just the presidential contenders or Democrat versus Republican overall.  Some qualify their rulings as far as sampling/selection bias while others take the rulings at face value.

Straight Politi-Quant websites.  The website Who's More Full of Sh*t (authored by J. Kalgreen) graphically rates the same....with an ongoing feed from PolitiFact.  It uses a score heavily weighted to the negative rulings.  It also gives an overall Democrat versus Republican rating.  While it's a pure rating website, its website set up is quite sophisticated.  As noted by Benjamin "Brash Equilibrium" Chabot-Hanowell, "I seriously wish I could code CSS and PHP as well as J. Kalgreen [WMFoS author] can."

While a great site, Who's More Full of Sh*t might not be linked by PolitiFact because its name might offend some.

Instead of saying who's more "truthier" or "honest" it goes in the opposite direction--scoring zero for True down to 200% for Pants on Fire.  So if my conservative counterpart decided to try to troll-comment the site (which he can't because no comments are enabled) he'd have to complain that PolitiFact's selection bias is the cause for Democrats being less full of sh*t, or for Republicans to be more full of sh*t.  Because, you see, both are still full of sh*t.

Malark-o-Meter is more of the nerdy Nate Silver variety of  meta/probability analysis and in that respect is like no other PolitiFact quantifier.  A newer website based on (I'm guessing) Joe Biden's outburst at the vice presidential debate, author Benjamin "Brash Equilibrium" Chabot-Hanowell is interested in mathematically determining a possible "bias" factor as well as the probability of how much each claimant is prone to lie. He includes the Washington Post Factchecker rulings in his analysis as well.

"Serious" Studies.  I've posted about and often referred often to Eric Ostermeier's 2010 study of 511 PolitiFact rulings  where he concluded with the infamous selection bias statement.

The Center for Media and Public Affairs did a study of both PolitiFact and Washington Post Factchecker rulings but did not really reach any other conclusion ("[took] no position on their accuracy") as to why Republicans score as less truthful than Democrats.

And of course, the website PolitiFact Bias did a research paper on PolitiFacts False/Pants on Fire rulings, coming up with a percentage factor that I often use in my blog.    There are others who have looked at the Pants on Fire, some in a humorous way, or a lot like PolitiFact Bias.

Website Articles.  One of the very first to quantify PolitiFact's Truth-o-Meter rulings was the pseudonymous "Cinematical" at the Daily Kos, back in 2009....and arrive at the conclusion as so many that would follow, that conservatives were less "honest" than liberals.  There's also been several scorings at the Daily Beast, as well as a nice quant review by the New York Times' FiveThirtyEight.com (Nate Silver's blog).

Blogs.  Anyone math-oriented knew the moment they saw the Truth-o-Meter that it begged to be measured.  I imagine there's a lot more than just those mentioned here. So (besides myself), here's three very good ones:

In 2011, I came across the blog Quibbling Potatoes authored by Steve, who had some great ideas for categorizations of the rulings that I now use for my data.  He also had a name for his score which he called...the Truth Index.  While I don't think PolitiFact took the name from Steve's site, one can certainly say Steve had a bit of clairvoyance.  It's obvious he put a lot of work into his charts and graphs and it would be nice to see him do it again.

Pandering Politicians is a blog (turned website like PolitiFact Bias) by someone who calls himself "PanderBear" and the Truth Index here is called the "Truth Quotient."  He has a lot of charts, especially time trendlines, and begs people not to even read chain e-mails because of their dismal "TQ."  And of course, in one of the comment sections we see the ubiquitous question from my conservative counterpart asking what he does about selection bias.  No answer from PanderBear.  At least as far as I could see.

Allen Allermann in a namesake blog entitled "Random Human Neural Firings" did a quantifying review about the same time as Steve called "Who do You Trust?"  He scored PolitiFact rulings in the opposite direction that many tend to do--he gave higher weight to the True and Mostly True instead of False and Pants on Fire.  He uses a "sports" analogy for what Brash Equilibrium terms the report cards of those rated by PolitiFact; he calls them "at bats."

So there we have, including my own, about a dozen attempts to quantify PolitiFact rulings.  Perhaps two qualifiers can be added: (1) None of these websites/studies/blogs are affiliated with or approved by PolitiFact (as far as I know) and (2) all numbers should be interpreted with caution.  And if you come across any websites/articles/blogs that quantify PolitiFact's Truth-o-Meter rulings, please let me know in the comment section.

Politi-Score: 2,788 Reasons Part 3

Doing calculations on PolitiFact (PF) rulings seems to be gaining in popularity. I noted a comment on the Facebook PolitiFact page the other day with regard to Ron Paul being the highest rated person on the PolitiFact’s truth-o-meter; however, I did offer a response that it was not Ron Paul but losing candidate for governor of Florida, Alex Sink. In fact, between Ron Paul and Alex Sink was the liberal public advocacy group moveon.org.

Just so you don’t have to use the link, the Politi-Score is calculated by assigning points to each ruling as follows:

Ruling	Number of points
True	100
Mostly True	75
Half True	50
Barely True	25
False	0
Pants on Fire!	-25

So, for example, if a politician has one True and one False ruling, that would be 100 for True and zero for False, totaling 100 points. Divided by two rulings, the Politi-Score averages to 50.

Getting back to Ron Paul and Alex Sink, here is a listing of everyone who had ten or more PolitiFact rulings from PF’s inception through the end of 2010, listed by highest to lowest Polit-Score. And yes, Michele Bachmann bottomed out the truthometer as the only one with a negative score (some Half Trues and Barely Trues from factchecks her 2011 SOTU response did put her into positive territory).

Politi-Score: Any Way You Shake It

During some recent internet research on PolitiFact, I came across something that answered what I had always wondered in regard to my Politi-Score: Is there someone out there that’s already done some thing similar to it? An article at Daily Kos published in October, 2009 by “cinematical” entitled “Analysis of PolitiFact: GOP/Conservatives Significantly Less Honest” had analyzed PolitiFact rulings and had done essentially the same thing: counted up the rulings and assigned a score to each ruling (scaled according to True, Mostly True, etc.), added them together and divided by the total number of rulings to get a “relative score.”

But the methodology was completely different than mine, in fact, almost the opposite, although it achieved something similar in its results as well as its conclusions.

“Limitations.” When I initially did the scoring, I wanted to get a relatively large sample, because when sampling a population (according to what I learned in Statistics), the larger the sample, the closer the results of what you are measuring should resemble that of the population. And I didn’t have a lot of time to go through individual listings (PolitiFact’s Personalities Page has over 800 individuals/ groups/blogs/e-mails listed), so I only counted those who had a lot of rulings. My cut off was to only include those with 5 or more rulings. Cinematical did the opposite. He did not include the scores of those who had 13 or more rulings. His reasoning was thus: “Inculsion [sp.] of these individuals would overwhelmingly weight their honesty over the collective honesty of the group.”

One of the commenters to this article said it well:

The fact that you limited the incidents to 13 per input person kind of skews it too much towards the "be kind" side. Why? Because the people with the most input likely have the most media presence. Therefore their propensity for truthfulness or lack thereof has a greater impact. An evaluation of the totality might prove interesting.