Quantifying the PolitiFact Quants

There's certainly a lot of people out there scoring PolitiFact rulings... I'm finding more and more of them every day.  Presented and scored in different ways, in studies, articles and blogs, I think it may be time to add them as a special section to my PolitiFact Detractors page, though "detractors" are not quite what they are.

While I hope I'm one of the better ones, I do need to make the case.  I have probably looked at these rulings in the largest variety of ways, and have done so on an ongoing basis for quite a while (since 2010, when I started this blog).  I've looked at the score in terms of writer, topic, local versus national, by individual compared with other fact-checkers, etc.   I'm looking for patterns.  For example, if Republicans lie more, do they lie about every subject?  (No, there are some subjects for which they are more truthful than Democrats)  Are lies more ideologically based? (Yes)  Do some PolitiFact writers appear to favor one party over the other based on the Truth Index? (Yes) Most of the other PolitiFact quants have looked at them in terms of just the presidential contenders or Democrat versus Republican overall.  Some qualify their rulings as far as sampling/selection bias while others take the rulings at face value.

Straight Politi-Quant websites.  The website Who's More Full of Sh*t (authored by J. Kalgreen) graphically rates the same....with an ongoing feed from PolitiFact.  It uses a score heavily weighted to the negative rulings.  It also gives an overall Democrat versus Republican rating.  While it's a pure rating website, its website set up is quite sophisticated.  As noted by Benjamin "Brash Equilibrium" Chabot-Hanowell, "I seriously wish I could code CSS and PHP as well as J. Kalgreen [WMFoS author] can."

While a great site, Who's More Full of Sh*t might not be linked by PolitiFact because its name might offend some.

Instead of saying who's more "truthier" or "honest" it goes in the opposite direction--scoring zero for True down to 200% for Pants on Fire.  So if my conservative counterpart decided to try to troll-comment the site (which he can't because no comments are enabled) he'd have to complain that PolitiFact's selection bias is the cause for Democrats being less full of sh*t, or for Republicans to be more full of sh*t.  Because, you see, both are still full of sh*t.

Malark-o-Meter is more of the nerdy Nate Silver variety of  meta/probability analysis and in that respect is like no other PolitiFact quantifier.  A newer website based on (I'm guessing) Joe Biden's outburst at the vice presidential debate, author Benjamin "Brash Equilibrium" Chabot-Hanowell is interested in mathematically determining a possible "bias" factor as well as the probability of how much each claimant is prone to lie. He includes the Washington Post Factchecker rulings in his analysis as well.

"Serious" Studies.  I've posted about and often referred often to Eric Ostermeier's 2010 study of 511 PolitiFact rulings  where he concluded with the infamous selection bias statement.

The Center for Media and Public Affairs did a study of both PolitiFact and Washington Post Factchecker rulings but did not really reach any other conclusion ("[took] no position on their accuracy") as to why Republicans score as less truthful than Democrats.

And of course, the website PolitiFact Bias did a research paper on PolitiFacts False/Pants on Fire rulings, coming up with a percentage factor that I often use in my blog.    There are others who have looked at the Pants on Fire, some in a humorous way, or a lot like PolitiFact Bias.

Website Articles.  One of the very first to quantify PolitiFact's Truth-o-Meter rulings was the pseudonymous "Cinematical" at the Daily Kos, back in 2009....and arrive at the conclusion as so many that would follow, that conservatives were less "honest" than liberals.  There's also been several scorings at the Daily Beast, as well as a nice quant review by the New York Times' FiveThirtyEight.com (Nate Silver's blog).

Blogs.  Anyone math-oriented knew the moment they saw the Truth-o-Meter that it begged to be measured.  I imagine there's a lot more than just those mentioned here. So (besides myself), here's three very good ones:

In 2011, I came across the blog Quibbling Potatoes authored by Steve, who had some great ideas for categorizations of the rulings that I now use for my data.  He also had a name for his score which he called...the Truth Index.  While I don't think PolitiFact took the name from Steve's site, one can certainly say Steve had a bit of clairvoyance.  It's obvious he put a lot of work into his charts and graphs and it would be nice to see him do it again.

Pandering Politicians is a blog (turned website like PolitiFact Bias) by someone who calls himself "PanderBear" and the Truth Index here is called the "Truth Quotient."  He has a lot of charts, especially time trendlines, and begs people not to even read chain e-mails because of their dismal "TQ."  And of course, in one of the comment sections we see the ubiquitous question from my conservative counterpart asking what he does about selection bias.  No answer from PanderBear.  At least as far as I could see.

Allen Allermann in a namesake blog entitled "Random Human Neural Firings" did a quantifying review about the same time as Steve called "Who do You Trust?"  He scored PolitiFact rulings in the opposite direction that many tend to do--he gave higher weight to the True and Mostly True instead of False and Pants on Fire.  He uses a "sports" analogy for what Brash Equilibrium terms the report cards of those rated by PolitiFact; he calls them "at bats."

So there we have, including my own, about a dozen attempts to quantify PolitiFact rulings.  Perhaps two qualifiers can be added: (1) None of these websites/studies/blogs are affiliated with or approved by PolitiFact (as far as I know) and (2) all numbers should be interpreted with caution.  And if you come across any websites/articles/blogs that quantify PolitiFact's Truth-o-Meter rulings, please let me know in the comment section.

Politi-Score: It may be His-[s]tory, but It’s He[u]r-istic

A college student by the name of Peter William Hurford published an interesting review with his own PolitiFact (PF) Politi-Score/Truth Index calculations. He titled it “Rating Presidential Candidates by the Polifact Heuristic.” While he’s spelled PolitiFact badly as Polifact, I’ve rarely heard the term heuristic. Anyway, Hurford adds the same type of graphics, although his analysis isn’t as sophisticated as my Truth Index/Politi-Score counterpart Steve at Quibbling Potatoes.

In “mass communication” a heuristic is defined this way:

In the study of media effects, judgmental heuristics have been shown to play an active role in the simplifying of news and political communication. Use of these cues and other signals from elites allow average people the opportunity to achieve a modest level of rationality in reaching a decision. This can be accomplished without having to devote any significant measure of cognitive effort normally required to arrive at thoughtful and considered choices. The limited capacity theory model and other information processing models have been influential in the study of how people encode, store and retrieve political information. Most people maintain a minimum level of interest in public affairs, and therefore employ simplifying shortcuts to arrive at political judgments. Common examples include referring to the complex military and intelligence activities by NATO forces in the Middle East simply as “the war on terror,” a reversal of a specific policy or position as a “flip-flop,” and the homogenization of any type of broad government assistance program as “socialism.”

In other words, PolitiFact acts as a “cue” or “signal” with its Truth-o-Meter (and its popular brand “Pants on Fire”) to simplify news and political communication.

What Hurford does is basically work up to the weighted average as employed by the Truth Index, only assigning double the value of the Truth Index:  instead of 150 points for Pants on Fire, or 1.5, Hurford assigns 300 points or 3.0, by 100-point (or 1.0) increments between each instead of 50 points (or .5). He starts with some of the presidential candidates and looks at the False and Pants on Fire as a percentage, then looks at the True and Mostly True as a percentage, and then adds other political figures, columnists and pundits who’ve been frequently rated by PolitiFact. He finally gets into a weighted average assigning his own scores (as described above).

His conclusions:

First, comparisons may be invalid because of statement sample biases. Again, it may be that 90% of every statement made by Glenn Beck is trustworthy and accurate, it’s just that of the specifically interesting things he says, his statements appear to be largely untrustworthy and inaccurate.

Second, comparisons may be invalid because of person inclusion biases. It certainly does look like Democrats appear seem to be more truthful than Republicans, but this probably isn’t true because of the large number of currently campaigning Republicans placed next to the large number of non-campaigning Democrats, and the propensity of people to utter falsehoods while under the pressures of the campaign trail and not in other locations.

Third, generalizations may be invalid because of an insufficient sample. It certainly looks like columnists (George Will and Paul Krugman) are far more trustworthy than pundits (Bill O’Reilly, Glenn Beck, and Rachel Maddow) and politicians (nearly everyone else), but the fact that we only included two famous columnists out of hundreds might mean that we have an unrepresentative sample, and columnists are largely inaccurate.

So what does this add up to?

When we acknowledge that we’re using a heuristic, and thus may not have accurate results but instead results that we consider accurate enough, we might not want to consider supporting or voting for Michelle Bachmann or Herman Cain on the basis of their continual habit of lying or being misinformed.

Hurford's "Politi-Scoring"

At the left is the weighted average "Truth-Score" Hurford came up with for individual politicians/ pundits/ columnists.  The results are somewhat similar to mine.  He lists them least truthful to most, with the names in red font being the Republicans, the blue Democrats.  At the bottom of this post I've put my most recent individual 2011 Truth Index calculation for comparison.  There's also one I did previously encompassing all PF rulings, not just those for 2011.

My conservative counterpart begs to differ, of course, in that he believes Hurford has made the error of “trusting elite opinions.” As I have gathered PF’s rulings statistics, I’ve made it a rule to qualify individual truth indexes by noting that a few rulings don’t mean a lot—what Hurford would term “an insufficient sample”. My last individual ruling rating was for those with 15 rulings or more, because I also believe that any conclusions reached should be based on “results that we consider accurate enough.” In addition, comparing those rulings to similar rulings done by other fact-checkers for the same statements provides a certain amount of validation to the accuracy of those results. All biases are not equal.  That's the reason for what I call the "Politi-Score."

Finally, the PolitiFact Truth-o-Meter rulings as quantified offer other statistical categorical “heuristics” beyond individual on a macro level, such as those critiqued in this blog and by Steve at Quibbling Potatoes: calculating each PolitiFact state’s  heuristic average, or a local versus national heuristic, one by subject, office-holder versus non-office-holder or month-to-month  as related to those political variables which might be affecting it. In other words, while individual rankings may be where the most interest is, we shouldn't consider them until there IS a sufficient sample to do so.  Given the quantity of rulings, these other areas of interest can be looked at in the mean time.

This chart only includes 2011 PF rulings on politicians (Rick Perry would be lower if ALL his rulings [pre 2011] were
included, so would more closely approximate Hurford's "heuristic" ratings).

Truth Index Trends: 2011 Source and "Occupation"

There’s been a variety of ways that other PolitiFact (PF) Truth Index quantifiers have broken down their results: about two years ago, someone named “cinematical” at Daily Kos was probably one of the first to talley up PolitiFact rulings and assign a score for each Truth-o-Meter ranking. He broke it down by office holders and non-office holders, groups and individuals, politicians and non-politicians.

My Truth Index counterpart, Steve of Quibbling Potatoes, has done one of the most comprehensive and analytical reviews of the PolitiFact Truth-o-Meter scores ever. His “Truth Index” moniker was copied by PolitiFact (although I don’t know if they were aware that Steve had preceded them). One of his breakdowns is by “occupation”—Activist, Advocacy Group, Journalist, Party Booster and Politician.

So I’ve decided to do it by what I’ll call “Source and Occupation":

Office-holders. This includes current, former and “potential”, for example, Herman Cain, since he is running for president but never held office.

Non-office holders. That’s everyone else, including journalists, blogs, and chain e-mails and as Steve might call them, just plain “rabble-rousers.”

Groups. I lumped advocacy groups and “party boosters” (like the RNC or DNCC) all together. Blogs and chain e-mails were not included here.

Individuals. This one is self-descriptive.

Media. That would include all journalists, pundits, and TV/radio talk show personalities. Someone who is a consultant to media, like Sarah Palin, would still be categorized as an office-holder since she was governor of Alaska.

Below is the breakdown (for the first nine months of 2011) of these categories by overall Truth Index score (multi-color), the by Democrat (blue), Republican (red) and all other political affiliations (light purple).

Going up from the horizontal zero line is “more truthful” and going down from it is “less truthful.” To keep the graph less busy I have not included a Truth Index number label for “all other.”

“Safety in numbers” appears to hold true when it comes to not telling the complete truth, as Groups appear to lie the most, which include “party boosters.” Steve had this to say about “party boosters” in his review of his compilation of the same database:

…party boosters (basically any organization with "Democrat" or "Republican" in its name, or such an organization's spokesman), whose sole function is to get folks to elect Democrats or Republicans, seem to lie at a shockingly high rate independent of party affiliation.

The Truth Index for Individuals appears to mimic office holders closely, since the majority of individuals are office holders. The score of non-office holders includes the media and the groups, as well as bloggers and chain e-mails, which made its score much lower than office holders across the board. Media personalities, however, constituted the biggest divergence in the Truth Index, with a variance between Republicans/Conservatives and Democrat/Liberals of almost 60 points. About 40% of all the statements coming from Republican/Conservative pundits or journalists were either False or Pants on Fire on the Truth-o-Meter; it was half that for the Democrat/Liberals.

Bloggers and chain e-mails by far are the worst offenders on the Truth-o-Meter. Their combined score (regardless of affiliation) was an almost Pants on Fire -133.33. There were six overly similar rulings of Pants on Fire done by various PF states all at the same time which concerned the ideology of certain Democrats, however, that I have a very strong inclination to do a separate post on. Nevertheless, nearly three fourths of the rulings on these two were Pants on Fire.

For 2012, I am thinking of separating office-holders to “local” and Washington-based as Steve did in order to contrast the two. He also did something interesting which was to look at senators as opposed to congressmen. At first I thought this should not be that relevant, but when you think about it, does the six year term as opposed to two year have any effect on “truthiness”? In a way, it might make you less inclined to be more truthful since you have the benefit of a longer term…or it could make you less? Hopefully I can continue where Steve may have left off and try to remain up to his excellent standard.

Politi-Score: Quibbling Potatoes, Very Well Done

Quite a cerebral Quibbling
Mr. Potato Head

I knew there had to be another blog, somewhere, someplace,  that was doing the same "Politi-Scoring" that I was. At least in the realm of assigning values to the PolitiFact (PF) Truth-o-Meter ratings and averaging them between Democrats and Republicans. After all, a few mainstream websites had been making some attempts at quantification, so it was an open idea that had been tried different ways many times.

I can’t recall exactly how this blog came up in a search result, but Quibbling Potatoes, a blog hosted by someone named Steve from Norman, Oklahoma, was doing almost everything I was doing, presenting similar graphs and quantifications of what became the PolitiFact Truth Index. He delved into some of the same issues with PolitiFact, and reached many of the same conclusions.

But the most interesting thing about Steve’s version of the Truth Index? I’ll put it in Steve’s words. 

This week PolitiFact announced a new feature on their mobile phone app...

They developed a chart of general truthiness against time so that trends could be seen more easily. ..

The feature is almost exactly like my PolitiFact Truth Index.

What did PolitiFact call this new feature? The PolitiFact Truth Index.

Coincidence? Who knows.

(Personally, I think it's more than a coincidence. But I can't prove that.)

As I said, Steve reached some of the same conclusions: that Republicans lie more than Democrats, that the lying increased, for example during the 2010 elections and the February Wisconsin protests, that the PF states have differences to PF National. I find Steve’s analysis more penetrating than my own, as I just publish what I find (or as he said in one of his posts, “charts ‘em as [he] sees ‘em”) and tend to leave more on conclusions to the readers. His post on subject analysis was excellent. Like me, he seems to be searching for a better way to graphically present his findings…yes, those murky clippings of Excel graphs (which is what his appear to be) aren’t the best. I appreciate his calculating the standard deviation but I’m doubtful how many readers have taken the necessary statistics to understand it.

But most of all, poor Steve was getting tormented by my conservative counterpart in his comments area, reaching the same impasse with him as everyone else. Steve referred to the blog PolitiFact Bias in one of his most recent posts: (emphasis added)