2013: First Six Months' Fact-checking for the Big Three

PolitiFact has definitely "downsized" in the U.S. even with its recent international presence in Australia.  At least for the time being.

 

Since 2011, I've generally expected 475 to 500 rulings from PolitiFact for each quarter.  When 360 were produced for the first quarter of 2013, I attributed that to the writers taking a break after the 2012 elections.  But the second quarter was pretty much a mirror of the first, going from 361 to an almost equal number of 363 rulings.   The dropping of PolitiFact Tennessee and no rulings from PolitiFact New Hampshire would not be enough to make up the difference.  Basically PolitiFact has cut back 20 to 25 percent.

 

With fewer rulings to record, I thought I might try something different:  back in 2010 I took a look at FactCheck.Org in and of itself.  I've compared rulings on the same statements many times, and on an individual basis, I've looked at various politicos in terms of Politifact, Factcheck.Org (FCO) and the Washington Post FactChecker.

 

This time I decided to record all three of the most prominent political fact-checkers (PolitiFact, WaPo FactChecker and FactCheck.Org) for the first six months of 2013.  I also tried to make a comparison in terms of the Truth Index.  Between PolitiFact's Truth-o-Meter categories and the Gepetto Checkmark to Four Pinnochios (a "whopper") used at the Washington Post, it wasn't too difficult there.    Factcheck.Org (FCO), however, does not rate fact-checks  with Truth-o-Meters or Pinnochios like the other two.

 

Just like it did when I reviewed it in 2010, I found that FCO generally did not fact-check "truthful" statements, as the great majority of its findings are negative: either misleading or false (It appears the Washington Post FactChecker leans that way as well, although not nearly as much).   It was a very subjective exercise for me to attempt to "rate" them as well, so the "Truth Index" I obtained, in my opinion, should not be construed as entirely accurate.

 

For the most part I based my FCO "ruling" on the same criteria as PolitiFact.  "Misleading" statements which contained an element of truth "but ignore[d] facts which would give a different impression" were categorized as "Mostly False" while statements which were "inaccurate," or incorrect (or when the FCO article called the statement a "false claim," which occurred fairly often) were categorized as False.    There were no "Pants on Fire" or "Whoppers" for FCO.

 

The results I obtained can be shown in the chart below.

The two "smaller" fact-checkers tend to concentrate on facts likely false based on the "Truth Index."

Even with PolitiFact's downsizing, it is still the dominant fact-checker of the "Big Three" with over 3/4 of all rulings for the first six months of 2013.    As noted, it checks a lot more statements which, as can be seen are (1) "True" and, due to its state operations, (2) statements by people with no political affiliation.

PolitiFact had a 57 percent higher overall Truth Index than the Washington Post FactChecker, and it was 82 percent higher than Factcheck.Org.

Truth Index+: Buh-Bye, Bachmann

What can I say.

Michele Bachmann was someone I termed a statistical "outlier" when it came to the PolitiFact Truth Index+, because she was so far off everyone else in terms of it, as well as in terms of its category Pants on Fire. When she announced she would not run for her congressional seat in 2014, PolitiFact "honored" her by describing the reasons for such a feat:

She had a remarkable streak: Her first 13 ratings were False or Pants on Fire.

Bachmann went on to compete in the 2012 Republican primary for president. She earned her first True rating for a statement she made during a 2011 debate that then-Sen. Obama "refused to raise the debt ceiling because he said President Bush had failed in leadership."

When a question was raised about her veracity, Bachmann cited our work during one of the many Republican presidential debates. "After the debate that we had last week, PolitiFact came out and said that everything I said was true," she said at a debate in Sioux City, Iowa, on Dec. 15, 2011.

Alas, she was totally wrong. In fact, the week before we had rated her statement that Mitt Romney instituted "socialized medicine" in Massachusetts as Pants on Fire.

Bachmann has been extensively covered on this blog--in fact, my very last post--and has been subjected to the Politi-Score, that is, taking into account all the fact-checks done on her by every source. In her case, she had four (4) sources because of the Minnesota Public Radio Fact-Checker in her home state. And she required more work than the others I've measured using the Politi-Score, as I ended up with a 5-part series when I covered her in August of 2011. It is still worth checking out, as it goes over many of the statements she made which were checked multiple times: (at that time) of the 50 statements from the 4 fact-checkers, a third of them were duplicated, and two statements were checked by all four fact-checkers (and for the most part rated quite consistently, which says something about the integrity of the fact-checkers' processes).

 

61% of Bachmann's statements
were rated False, of which
42% were Pants on Fire

Her Truth Index+ alone, however, should be reviewed in proper context. PolitiFact provided her last ratings by percentage for 59 rulings (see chart at right), but that was all (the same they would do for anyone else). Here is how she looks (below) when you calculate Bachmann's Truth Index+ and compare it to the overall Truth Index+ (through 3/31/2013), as well as that for Republicans overall, for Democrats overall, and compared to her fellow Republicans in Congress, and as an added individual comparison, to the next best female Democrat in congress who probably gets closest to her in terms of making those occasional outrageous statements, Nancy Pelosi, although she's only been subjected to the Truth-o-Meter 22 times, I'm assuming because she relinquished her Speaker of the House position following the Republican mid-term election rout in early 2011(and didn't attempt a run for president as Bachmann did).

 

As can be seen, the Truth Index+ for all the other measures goes from 84% higher for the Democrats overall down to 54% higher for her fellow Republicans in the House of Representatives. I'd also venture to say that for those who've had 50 or more rulings, which puts her in a very special Truth Index+ category especially for females, she probably ranks dead last as far as the Truth Index+ (males AND females).  And the more rulings, the more tell-tale "for truthiness" is the Truth Index+.

 

Today, PolitiFact published an article re-iterating its "principles" with regard to doing rulings. Bill Adair, PolitiFact's creator and outgoing manager, stated under the heading "Choosing Claims to Check" that "we select the most newsworthy and significant ones" and in a bulleted list of "questions they ask themselves" in selecting a statement asked "Would a typical person hear or read the statement and wonder: Is that true?" And Bachmann certainly met that criteria in many of her statements.  

 

In some ways, hard-line conservativism and Tea Party associations made her a colorful character.  But I can't really say she will be missed. Well, we still have 19 months to go, and anything can happen...so I'll keep my mouth shut.

PolitiFact: A Truth Index Video!

2012 in Review at PolitiFact (Continued)

A few items have been posted already about the year 2012 at PolitiFact (PF).  It was their biggest year because of the presidential election, with 1,961 rulings produced.    Here's how it looked for all Truth-o-Meter categories as a percentage, in 2012:

Click to enlarge

In terms of Republicans versus Democrats, Democrats had a 2012 Truth Index+ of 58.5 (-3.7 PF Truth Index) for 735 rulings, while the Republicans had a 49.8 (-25.6 PF Truth Index) for 1,078 rulings.  That means that for the statements fact-checked by PolitiFact, the Democrats' rulings averaged about 17.5 percent higher (in proportional terms of Truth Index+).

Click to enlarge:  Truth Index+ for party is in parentheses

We've looked at month-to-month already, and took at a better look at Pants on Fire.  Let's take a look at the top ten "rulees"--those who were fact-checked.  This may explain to some degree why more Republicans are checked than Democrats.  While Obama was the person most fact-checked, he was the only Democrat on the top ten list.  Outside of people, only chain e-mails had enough rulings to get to the top ten.  The PolitiFact states showed their contributive power, with Chris Christie (PolitiFact New Jersey), Rick Scott (PolitiFact Florida), and Scott Walker and Tommy Thompson (PolitiFact Wisconsin) making it to the list.  Other than that, there were the expected presidential candidates (Vice president Joe Biden was Number 13 with 14 fact-checks).  The top ten shown reflects about 25 percent of all rulings; it should also be noted that of the 760 individuals/organizations "politi-facted" in 2012, 500 of them only received a single ruling (also about 25 percent of the rulings).  This shows how PolitiFact concentrates its efforts.

Click to enlarge:  Chain e-mails have always had one of the lowest Truth Indexes (highlighted).

In terms of topic, there is some overlap, that is, PolitiFact assigns multiple subjects by the nature of the fact-check, so these aren't quite as straightforward.  Generally, this is the "first" subject assigned in the fact-check, although it can be a subjective selection.  Occasionally it may be changed if I feel it's not correct, but that is rare.  Nevertheless, here are the top ten subjects which cover about 57 percent of all the rulings.  There's actually 11 because Deficits and Abortion tied for tenth place with 56 rulings each.

Click to enlarge:  Healthcare had lowest Truth Index because of high number of False and
Pants on Fire Obamacare claims (highlighted)

As far as separating Democrats from Republicans in other categories, I may be doing that as the year progresses, but more likely in terms of all rulings as PolitiFact quickly approaches 7,000.  Hold that dial.

 

PolitiFact 2012 Truth Index Outbreak Confirmed

Following the election, it was apparent from Facebook posts that PolitiFact had gone into "clean up" mode and had taken a hiatus from fact-checking.  They needed the rest after kicking it up to about 238 rulings in October, about 40 percent more than their usual output.  It updated its Obamameter of promises, and  only two, yes, one plus one, rulings were published between the election and the new year.

 

Because of those two rulings and the PolitiFact states continuing to churn out  rulings at a lower pace, PolitiFact's 1,961 rulings for 2012 exceeded their production for all other years (2011 was 1,919, second place).  If it continued at same rate as in the first ten months of 2012, there would have been over 2,100 rulings for 2012.  Through the end of 2012, since PolitiFact began in 2007, it has done a total of 6,658 rulings.
 



PolitiFact's Truth Index App has
rather limited info.

As usual, Republicans comprised 55 percent of the rulings in 2012; the Democrats only 37.5 percent.  The congressional majority and the predominance of Republican legislatures and governors in PolitiFact states Florida, Georgia, Virginia, Ohio, New Jersey, Texas and Wisconsin contributed to the Republican numbers.  PolitiFacts New Hampshire, Oregon and Rhode Island, while having more Democrats in power, do not contribute as many rulings overall as the other seven.   Those three states combined contributed 299 rulings for 2012, compared to 1,170 for the other seven heavily Republican states.

 

Recently PolitiFact noted in an article that since the election, the Truth Index had an "outbreak" up the "truthiness scale" in its week-to-week graphic that I'm assuming is available on its android application (I don't have a smart phone because in my circumstances I really don't need one).  Using PolitiFact's Truth Index methodology, the increase in "truthiness" is pretty striking as shown here, smoothed out month to month for the entire year of 2012.  The only other difference to PolitiFact is, it's being shown for Democrats (blue line) versus Republicans (red line), with the grand average as the dotted purple line (which includes those with another or no political affililiation).   The number of rulings published  is shown in parentheses next to the month.

Click to enlarge:  The Truth Index went into a narrow range between parties in the 3
months prior to the election.  Note this is not Truth Index+, but PF's methodology.

The chart has what one might call a "bowl" pattern bottoming out in July; after bounding back in August, the Democrats and Republican resumed a rather closely corresponding line with both pivoting up sharply following the election.  So, yes, this confirms PolitiFact's breakout.
 

The question about this chart that most may wonder about:  what caused the Republicans to "bomb" in July?  The percent of False and Pants on Fires of the total rulings for that month was the highest for the year, at 38.4 percent.  A quick check on Pants on Fire revealed the most likely answer.  July was the "month" of the ruling that received the most votes from PolitiFact readers as the 2012 Lie of the Year, although it was not selected for that "prize" by its editors.  It was the statement about Obamacare being the biggest tax increase in history.  It was repeated that month four times, along with another Obamacare myth about a 3.8 percent real estate transaction tax (ruled on twice).

Finally, what happened to PolitiFact Tennessee (as of January 29, 2013)? They haven't done a ruling since late November, and they are not on the state drop-down at the PolitiFact website.  Searches don't seem to be turning anything up with any sort of "announcement."  Well I hope PolitiFact Tennessee doesn't just waltz away.

Truth Index+: Office Holder Breakdown

The final post for 2012 will be an office holder breakdown from all PolitiFact rulings through the eve of the 2012 election (November 5, 2012).  The majority (about 53%, or 3,450) of statements put to the Truth-o-Meter at PolitiFact for fact-checking are from those in office:  from the president to senators and representatives, all the way down to local elected officials, such as mayors and city council members.   Here is how the official "pie" is cut:

Click to enlarge

Office holders as a group have consistently held a significantly higher Truth Index (14.7% higher overall) than that of those who were not in office.  For example, they had 43% more statements rated True, and 28% more statements rated True and Mostly True.  In the "subjectively rated" Pants on Fire category, office holders received just 45 percent of the Pants on Fire ratings percentage of those who did not hold office (5.4% versus 12.1%).

The reasons for this difference may be that a large portion of those "not in office" were advocacy groups and PACs, those without a "name" only seeking to promote a cause, and would not lose their voice in losing an election.  Pundits and chain e-mails, mostly in a position of "playing to the choir" could risk making ridiculous statements.

For Democrats versus Republicans, there's no Truth Index comparison for president/vice president: for most of PolitiFact's existence, the office of president has been held by a Democrat.  George W. Bush was only rated once prior to the election of 2008; PolitiFact preferred to concentrate on candidates Obama (168 rulings) and McCain (155 rulings) in 2007 and 2008.  So the chart below does not show any office of president comparison.

Click to enlarge:  Closest party to party Truth Index+ average was governors.

The (raw) Pants on Fire factor, a measure of how much more often a party is assigned a Pants on Fire when a statement rated by PolitiFact is determined False, shows a slight advantage for the Republicans overall and specifically in the categories of both houses of congress, governors and local governments.  The red (Republican) and blue (Democrat) bars are all the same--they all equal a one, with the "fire" bar a measure of how much more that party is assigned the Pants on Fire when they get a False.

Since there was no comparison for president (as I explained above), I decided to compare Obama to Romney for this chart, although Romney, since he was not in office, was not included in the 3,450 rulings.  It does show, however, quite a substantial variance between the two:  Obama has kept a very low percentage (2 percent overall) of Pants on Fire.  It may be a reflection of his very careful speech or maybe something to do with PolitiFact. 

Truth Index+: By Occupation

Early in 2012, using PolitiFact's own Truth Index calculation method, I published a Truth Index by occupation for the year 2011.  That was for 1,919 PolitiFact rulings. This time it will be for ALL rulings through the 2012 election, that is, 6,535 rulings.  The result is somewhat similar to that for the 2011 analysis.  Here is how they break down by category:  

Click to enlarge:  Each proportional slice of the pie gives the number of rulings

 This time I am using the Truth Index+ method to calculate the results as applied to the 6,535 rulings, which equally measures (starting at zero for Pants on Fire to 100 for True) all Truth-o-Meter categories.  Office holders, of course, compose the largest group of PolitiFact "rulees" comprising over half.  This time, however, I've decided to separate out the "candidates" who were not in office at the time they made their fact-checked statements.  This greatly reduces the "Other" category which is where they would have gone.  The majority of candidates in this category were Republican: for example, Mitt Romney, Herman Cain, and from 2008, Rudy Giuliani and Fred Thompson.   In fact, there were only 22 statements in this category attributed to Democrats, which means any comparison between Democrats and Republicans in terms of the Truth Index means very little.

 

The Truth Index+ for those categories shows the disparity between Republicans and Democrats attributed by some to "bias."  Every category except one (PACs and Super PACs) had an index favoring the Democrats.  In the case of media (pundits, chain e-mails, blogs, etc.), the variance was quite large, at almost 40 percent, compared to the overall average variance of 16 percent, or 15 percent for those holding office.

 

Click to enlarge:  Media (chain e-mails, pundits, celebrities) was by far the least truthful.

Finally, here's the raw "Pants on Fire" factor graphed for the same occupation categories.  This factor is a measure of how much more often a party is assigned a Pants on Fire when a statement is determined False by PolitiFact.  The red (Republican) and blue (Democrat) bars are all the same--they all equal a one, with the "fire" bar a measure of how much more that party is assigned the Pants on Fire when they get a False.    Even with the Democrat dominance of a favorable Truth Index, this factor does not correlate that well with it, except in the case of the media.  The "presidential candidate" factor is not a good one, either, because as noted above there were so few Democrat rulings (of the 467 rulings, 22 were on Democrats.)  And it gets even more out of kilter when the "in office" number is broken down further by position, up next.

Click to enlarge:  While media had worst Truth Index, advocacy groups (Republican) told most "whoppers."

 

Truth Index+: The Top Ten PolitiFact Writers

There are ten PolitiFact (PF) writers who have produced just about half of PolitiFact's rulings, from its beginnings in 2007 up until I ended my compilation on Election Eve, 2012.  Another roughly 160 writers composed the rest of them, many serving multiple functions of writer, researcher and editor.  There were 47 writers who wrote just one to two rulings each.

 

The highest producer by far is Lou Jacobson, whose 676 fact-checks (some done in association with other writers at PolitiFact) make up a bit over 10% of PolitiFact's total output.  Number 2 is Angie Drobnic Holan, now the Editor at PolitiFact Florida, at 571.

 

Below is the Truth Index+  for the Top Ten, with scores for Republican versus Democrat.  The best way to look at these averages in terms of "bias" or "lean" would be the percent variance in the Truth Index+ between party for each writer.  Lou Jacobson, Angie Drobnic Holan, Robert Farley and PF Georgia's Willoughby Mariano have about a 22% variance leaning toward the Democrats, only to be exceeded by Number One C. Eugene Emery of PolitiFact Rhode Island, who has a whopping 38% Democrat lean.   Tom Kertscher at PolitiFact Wisconsin swings 22% in the opposite direction; the remainder go from 3% in favor of the Republicans to around 10% in favor of the Democrats.

 

Click to enlarge:  NA in parentheses = PF National; then number of rulings written.

So, what can we learn from the Truth Index+ for each writer?  I decided to take a closer look at the top four writers and an "outlier" in terms of some of the items I had keyed in:  party, titles, people, subjects.

 

Lou Jacobson, for example, was No. 5 in "lean" variance as noted above, although other than Emery he was in a virtual tie with four others.  But he had a few features in his 676 rulings that stood out:  he is definitely "the economics guy" on the PolitiFact team.   The subject of about 40% of his rulings was a fiscal one: economy, federal budget, stimulus, taxes etc..  This was a far larger percentage than any other writer.  Healthcare was the subject of 25% of his rulings.    In terms of "occupation" his being on PolitiFact National's team meant that about 30% of his rulings were on the president and vice president and the presidential/vice presidential candidates (in 2008 and 2012).

 

A check on names revealed a notable person who seems one of Jacobson's "fact-check favorites"--Michele Bachmann.  He did almost a third (17) of her 54 rulings, and gave her 5 of her 12 Pants on Fire awards, as well as five Falses (in other words, when a claim was found False for Bachmann, Jacobson determined it was Pants on Fire half of the time.)  In comparison, Angie Drobnic Holan only gave Bachmann one Pants on Fire out of five rulings found False.

Truth Index+ by PolitiFact National and States

The state by state Truth Index+ has held pretty consistently since I first started crunching the data last year (even using the PolitiFact calculation).  Now I can present it (with a great deal more accuracy) for all rulings since 2007 through the eve of the 2012 general election on November 6.

 

Click to enlarge

That trend holds overall with one exception: that the Truth Index+ for Democrats exceeds that of Republicans.  Overall the variance is 15.9%.  To express it appropriately, the statements selected for Democrats by PolitiFact National as well as all state PolitiFacts have been on average rated about 16% higher in terms of more True/Mostly True/Half True(s) and fewer Mostly Falses/False/Pants on Fire(s) on PolitiFact's Truth-o-Meter.  Also, if you designate the Pants on Fire as equivalent to False instead of rating it lower (that is, remove the effect on the Truth Index score of Pants on Fire), this variance decreases to 13.9%.

 

At the state level, the Truth Index+ variance between Republicans and Democrats ranges from a nearly dead even 1.4% at PolitiFact Virginia, to 27.7% at PolitiFact Tennessee, although there it may be an anomaly due to the fact it had the lowest number of rulings measured (73 total) as the newest PolitiFact state franchise.  And the exception is PolitiFact Wisconsin at a "negative" 7.2%, that is, their Truth Index+ is the only one favoring the Republicans.

 

The anti-PolitiFact propaganda website PolitiFact Bias (PFB) recently posted its 2012 Pants on Fire factor (how much "more" it claims Pants on Fire are assigned to Republicans) by PolitiFact state.   And it has discovered via the results it shows, as I've reported before, some of the states actually assign more Pants on Fire to Democrats.

 

In its original research paper, we are told that the factor is adjusted for chain e-mail and "intra party" claims.   Because of the primaries this year, there were a lot of such "intra party claims", and due to that, although I go in the general direction of PFB's calculations of the factor, I was not able to match it.

Quantifying the PolitiFact Quants

There's certainly a lot of people out there scoring PolitiFact rulings... I'm finding more and more of them every day.  Presented and scored in different ways, in studies, articles and blogs, I think it may be time to add them as a special section to my PolitiFact Detractors page, though "detractors" are not quite what they are.

While I hope I'm one of the better ones, I do need to make the case.  I have probably looked at these rulings in the largest variety of ways, and have done so on an ongoing basis for quite a while (since 2010, when I started this blog).  I've looked at the score in terms of writer, topic, local versus national, by individual compared with other fact-checkers, etc.   I'm looking for patterns.  For example, if Republicans lie more, do they lie about every subject?  (No, there are some subjects for which they are more truthful than Democrats)  Are lies more ideologically based? (Yes)  Do some PolitiFact writers appear to favor one party over the other based on the Truth Index? (Yes) Most of the other PolitiFact quants have looked at them in terms of just the presidential contenders or Democrat versus Republican overall.  Some qualify their rulings as far as sampling/selection bias while others take the rulings at face value.

Straight Politi-Quant websites.  The website Who's More Full of Sh*t (authored by J. Kalgreen) graphically rates the same....with an ongoing feed from PolitiFact.  It uses a score heavily weighted to the negative rulings.  It also gives an overall Democrat versus Republican rating.  While it's a pure rating website, its website set up is quite sophisticated.  As noted by Benjamin "Brash Equilibrium" Chabot-Hanowell, "I seriously wish I could code CSS and PHP as well as J. Kalgreen [WMFoS author] can."

While a great site, Who's More Full of Sh*t might not be linked by PolitiFact because its name might offend some.

Instead of saying who's more "truthier" or "honest" it goes in the opposite direction--scoring zero for True down to 200% for Pants on Fire.  So if my conservative counterpart decided to try to troll-comment the site (which he can't because no comments are enabled) he'd have to complain that PolitiFact's selection bias is the cause for Democrats being less full of sh*t, or for Republicans to be more full of sh*t.  Because, you see, both are still full of sh*t.

Malark-o-Meter is more of the nerdy Nate Silver variety of  meta/probability analysis and in that respect is like no other PolitiFact quantifier.  A newer website based on (I'm guessing) Joe Biden's outburst at the vice presidential debate, author Benjamin "Brash Equilibrium" Chabot-Hanowell is interested in mathematically determining a possible "bias" factor as well as the probability of how much each claimant is prone to lie. He includes the Washington Post Factchecker rulings in his analysis as well.

"Serious" Studies.  I've posted about and often referred often to Eric Ostermeier's 2010 study of 511 PolitiFact rulings  where he concluded with the infamous selection bias statement.

The Center for Media and Public Affairs did a study of both PolitiFact and Washington Post Factchecker rulings but did not really reach any other conclusion ("[took] no position on their accuracy") as to why Republicans score as less truthful than Democrats.

And of course, the website PolitiFact Bias did a research paper on PolitiFacts False/Pants on Fire rulings, coming up with a percentage factor that I often use in my blog.    There are others who have looked at the Pants on Fire, some in a humorous way, or a lot like PolitiFact Bias.

Website Articles.  One of the very first to quantify PolitiFact's Truth-o-Meter rulings was the pseudonymous "Cinematical" at the Daily Kos, back in 2009....and arrive at the conclusion as so many that would follow, that conservatives were less "honest" than liberals.  There's also been several scorings at the Daily Beast, as well as a nice quant review by the New York Times' FiveThirtyEight.com (Nate Silver's blog).

Blogs.  Anyone math-oriented knew the moment they saw the Truth-o-Meter that it begged to be measured.  I imagine there's a lot more than just those mentioned here. So (besides myself), here's three very good ones:

In 2011, I came across the blog Quibbling Potatoes authored by Steve, who had some great ideas for categorizations of the rulings that I now use for my data.  He also had a name for his score which he called...the Truth Index.  While I don't think PolitiFact took the name from Steve's site, one can certainly say Steve had a bit of clairvoyance.  It's obvious he put a lot of work into his charts and graphs and it would be nice to see him do it again.

Pandering Politicians is a blog (turned website like PolitiFact Bias) by someone who calls himself "PanderBear" and the Truth Index here is called the "Truth Quotient."  He has a lot of charts, especially time trendlines, and begs people not to even read chain e-mails because of their dismal "TQ."  And of course, in one of the comment sections we see the ubiquitous question from my conservative counterpart asking what he does about selection bias.  No answer from PanderBear.  At least as far as I could see.

Allen Allermann in a namesake blog entitled "Random Human Neural Firings" did a quantifying review about the same time as Steve called "Who do You Trust?"  He scored PolitiFact rulings in the opposite direction that many tend to do--he gave higher weight to the True and Mostly True instead of False and Pants on Fire.  He uses a "sports" analogy for what Brash Equilibrium terms the report cards of those rated by PolitiFact; he calls them "at bats."

So there we have, including my own, about a dozen attempts to quantify PolitiFact rulings.  Perhaps two qualifiers can be added: (1) None of these websites/studies/blogs are affiliated with or approved by PolitiFact (as far as I know) and (2) all numbers should be interpreted with caution.  And if you come across any websites/articles/blogs that quantify PolitiFact's Truth-o-Meter rulings, please let me know in the comment section.

Truth Index+: The PolitiFact Saga on Obamacare

In the history of PolitiFact rulings, from its beginnings in 2007 to the election of 2012, the most covered subject was healthcare. PolitiFact currently has 42 pages of rulings related to healthcare. While there's a lot of overlap with other subjects (particularly fiscal ones) as designated in every ruling, my counts for 6,535 PolitiFact rulings show healthcare and Medicare were the "main" subject(s) for over 10 percent of them.

 

In 2007, the very first characterization of a Democrat national healthcare reform plan as "socialized medicine" which earned a False was from budding 2008 presidential candidate Rudy Giuliani. PolitiFact called it "pure hyperbole."   In early 2008, eventual 2012 presidential candidate Mitt Romney was the first person fact-checked for what would become the PolitiFact 2010 Lie of the Year--that "Obamacare" (the meme not yet invented for healthcare reform) was a "government take over" of the healthcare system.

Click to enlarge:  Low number of Democrat rulings caused larger swings

There were very few fact-checks (12) evenly divided between party on healthcare reform as it related to Obama's proposals in 2007 and 2008, compared to what happened after he became president. So the Truth Index+ at that time did not tell us much. Indeed, as I noted in a previous post on what caused the big increase in Pants on Fire assignments to Republicans, the year 2009 was a very big ruling year when it came to healthcare reform and Obamacare. In that year, over 1/4 of PolitiFact rulings related to that topic.

Obamacare, however, was the leading PolitiFact ruling topic in 2009: healthcare was the subject of over 30% of the rulings with about 27% relating to healthcare reform, and made up over 1/3 of the Pants on Fire. With the majority coming from Republicans, this was for the most part what drove up the Pants on Fire factor against them.

The year 2010 was not as important in terms of PolitiFact's concentration on Obamacare; it was a pivotal year, however, in terms of the Republicans doubling down on attacking it. Republicans dominated on dubious healthcare claims as 64 fact-checks brought their Obamacare Truth Index+ down to under 34. It continued into 2011 and worsened in 2012. The number of Obamacare-related rulings doubled from 2011 to 2012, when it seemed as if the Republicans thought the public dislike of it would get them the same type of election results as in 2010.

Truth Index+: Are Election Winners More "Truthful"?

Since PolitiFact has been through three national election cycles and a few minor ones, I've always been curious as to whether the fact-checking matters in regard to who gets elected. Or to put it another way, are those who win more "truthier" according to the Truth Index.... or less? So as an experiment, I went through the PolitiFact rulings and keyed in whether the candidate/official won or lost in a pending election from the time he or she made the statement.

But first I had to do a lot of filtering of my list of 6,535 rulings (all rulings through 11/5/12).  All the non-official people (like pundits and advocates) and groups (like websites, bloggers, chain e-mails, PACs etc.) had to go, of course. I also didn't include (or partially included) the rulings of those who retired or lost in the primaries (good example: Dennis Kucinich). There were some who won and lost, like John McCain: I rated his statements prior to his 2008 presidential election loss as a "loser" and those since as a "winner" since he won re-election as Arizona senator in 2010. Local and state elections are also excluded except for governors races. And just for the heck of it, I also keyed in those who were in the Tea Party caucus. When I got done, I was down to 2,937 rulings.

Click to enlarge:  Winners appear to make fewer False/PoF statements regardless of affiliation.

As of November 5, 2012, for all 6,535 PolitiFact rulings, the Truth Index+ is 54.2. For the 2,937 candidates making campaign statements, it was about 5 percent higher at 57. However, the "all" includes everything along with the kitchen sink. A better comparison might be to the Truth Index for those were in office at the time of the statement, which is similar to those facing elections, which was a tiny bit higher 57.7. Compared to the overall, however, what this says is that individuals and those in official positions tend to be more truthful on their own than groups or those who advocate for them. I've already shown this to be the case when I've posted calculations of the Truth Index separately for office-holders versus the various categories of advocacy groups, party boosters, media pundits, etc.

The "bias" factor would not be as relevant here because I am not looking at this in terms of Democrat versus Republican. But it's evident from these statistics that the "winners" are found more "truthful" by the PolitiFact fact-checkers than are the "losers" by about 8 percent in terms of the Truth Index+. It's about 6 percent if you re-adjust the Pants on Fire category to False.

Comparing Republicans and Democrats gets a wee bit trickier. Democrats have always held a higher Truth Index in general, and even though some say it's due to selection bias, something did emerge from the analysis of interest concerning the Republicans that can't be totally attributed to bias. While the Democrats had a pretty substantive variance between winners and losers (63.6 versus 55.8, or 14 percent), the Republican's variance was insignificantly small--53.5 versus 52.8, or about 1.5 percent. What this says is that the Republicans have about the same level of "truthiness" based on statements evaluated whether they win or lose. What this means, I don't know.

Truth Index+: Adjusting for Pants on Fire

The Pants on Fire factor as developed by anti-PolitiFact propaganda website PolitiFact Bias (PFB) leaves much to be desired--specifically, it does not take the other four rulings (True through Mostly False) into account.  Their reason is the more "subjective" nature of the Pants on Fire ruling as defined by PolitiFact itself (not just inaccurate but "ridiculous").  Basically it measures how much more (or less) Pants on Fire is assigned to Republicans when a ruling is false.  But when the Pants on Fire are taken out completely (and are calculated as False), the rulings still favor the Democrats on average about 17% (for all 2012 rulings through election day) using PFB's own baseline methodology (adjusting for intra-party, chain e-mails, pundits, PACs and party boosters).  Regardless, the Pants on Fire effect is very minimal, as recently pointed out by Brash Equilibrium's Malark-o-Meter and in an earlier review I did in 2011.

A better method might measure the overall percentage change in the (newly formulated) Truth Index+ when the Pants on Fire are removed and categorized with the same score as False.  In the example data above, the Democrats' Truth Index+ is improved 1.4% by eliminating Pants on Fire, while the Republicans Truth Index+ is improved 3.5%.  This percentage could be called the Pants on Fire Bias factor.

One could then either take the remaining Republican to Democrat difference in the Pants on Fire adjusted Truth Index, which in this case is 16.8%, as what would be the undefined or non-subjective "bias" variance.  One could also measure the difference between the 1.4% and the 3.5% which would be about 2.5.  This is a Pants on Fire bias factor which incorporates the other rulings.

In other words, in terms of ALL rulings True through Pants on Fire, the Republican Truth Index is effected 2.5 times as much by Pants on Fire than is the Truth Index of the Democrats.  All because Republicans are assigned Pants on Fire about 74% more than Democrats when a ruling is false (to use the PFB factor).

It's ironic that I came out to 16.8% since Brash Equilibrium's statistical testing of a sample of PolitiFact rulings hypothesized there may be as much as 17% variance in the rulings due to bias.  But that is a qualified "as much as" so it doesn't really tell us anything.  More to follow as I apply this to all rulings through election day, 2012.

Truth Index+: Obama-Romney FIVE MORE DAYS

Five more days...unless of course, something happens due to Super-Storm Sandy and voting gets delayed somewhere in New Jersey or New York City or New England....or Romney wins the vote but loses the Electoral College and sues...you never know.  The only "poll" I really pay attention to is one and even though it's now almost 80% favorable to Obama, that still means, because the fellow who calculates it used to do odds in Vegas, that, like any table in Vegas, the 20% can still come in.  And I'm very superstitious when it comes to being overly hopeful when gambling or making predictions.

But here's where Obama and Romney stand with regard to what was in the PolitiFact Truth-o-Meter count for them as of October 30, 2012:

 

 

Obama is about 22.8% more "truthful" based on PolitiFact rulings than Romney.  PolitiFact is 3.6 times more likely to elevate a False ruling for Romney to Pants on Fire than they are for Obama.

 

For the trend line, I'm going to add a period-to-period average (in the dotted line) along with the accumulated average (shown in the solid line).  This is so you can see where the Truth Index+ made any significant movements:

Click to enlarge:  Romney's 2011 28.6 Truth Index was from his presidential announcement (see below)

And what did Romney say that got his Truth Index down so low in early 2011?  Well, a few of the things he said then he continued to repeat, and stated during the October, 2012, debates.  He also re-stated one of PolitiFact's Lies of the Year:  that "Obamacare" was a government take-over.  He also made the comment about Obama's worldwide "Apology Tour" and that the United States was inches from being a free economy.

 

But it looks like--for the last 4 months, at least--that even here Romney has narrowed the gap on the Truth Index+.  Since June Obama rates only 12% more "truthier" than Romney with the PolitiFact rulings.  Of course, there are some that say this is all for naught because PolitiFact has a left-leaning bias.  But so what...because there are also some that say that's malarkey, and back it up.

Three Pants on Fires in a row didn't help Romney's Truth Index in 2011.

 

 

 

Pants on Fire Factor: Obama-Fact

The addition of the state PolitiFacts began with the appearance of PolitiFact Florida in 2009, but really didn't really get in gear with publishing a substantive number of rulings until 2010.  In fact, in 2009, PolitiFact "Florida" published a grand total of 9 rulings, bringing in lead PolitiFact Florida writer Aaron Sharockman around October.  Then things took off in 2010 as Texas, Georgia, Rhode Island were added in the first half and then Ohio, Oregon, Wisconsin and Virginia in the second half.

 

It would seem as if the states came along to fill in the gap of the Congressional Quarterly, whose writers worked with PolitiFact doing rulings from 2007 through 2008.  However, as I described in my original assessment of this group and what its "affects" may have been , the types of statements selected at that time were of a different nature:  in the case of Congressional Quarterly, they related to its area of expertise, congressional positions and votes (although they did do other types).  In the case of the PolitiFact states, however, many rulings had to do with matters of a local nature, for example, one of the first PolitiFact Florida rulings on  a statement by then-Governor Charlie Crist who was bragging up the reduced rates of a negotiated, un-subsidized Florida healthcare program called Cover Florida Healthcare (found False).

 

The "Pants on Fire factor" (PoF) as I call it, a measure of what is claimed to be PolitiFact's bias for rating a False ruling as Pants on Fire, is shown below as aggregated from 2007 to September of 2012.    It is a comparison of Democrats to Republicans, with a factor of 1.0 translating to mean that PolitiFact assigns Pants on Fire to Democrats at the same (or equal) rate it assigns Republicans.  Anything below the 1.0 means Republicans are assigned Pants on Fire less than Democrats, anything above the 1.0 means Republicans are assigned Pants on Fire more than the Democrats (as a percentage of False + Pants on Fire rulings).

 

 

The red line shows the growth of the factor from favoring the Republicans to favoring the Democrats (or growing ever more negative for Republicans) over PolitiFact National's six years of Truth-o-Meter rulings.  The chart begins in quarterly increments until the end of 2008 to show how the Congressional Quarterly scored for that period as well, displayed in the green line. Thereafter the increments are annual.

 

At the end of 2008, PolitiFact National (including the Congressional Quarterly) had a factor below 1.0, although as shown by the green line, the Congressional Quarterly (CQ) alone was assigning Pants on Fire to Republicans 42% more than to Democrats.  With CQ behind them, in 2009 the PoF factor made its biggest jump ever at PolitiFact National, from .90 to 1.60.    For the year 2009 alone, PolitiFact National's Pants on Fire factor was a whopping 5.06.

 

Accumulated through the end of September of 2012, it now stands at 1.94--in other words, in the course of its history, PolitiFact National assigned Pants on Fire almost twice as often for Republicans as it did for Democrats.

 

Combined with the state PolitiFacts, however, the chart takes a different slope, as shown by the purple line.  It starts at the same high point as in 2009 because of there being only 9 PolitiFact Florida rulings, but as they accumulated, the aggregate Pants on Fire factor for PolitiFact as a whole dropped, from the 1.60 in 2009 down to its current 1.33.  The states themselves, if they were separated from PolitiFact National, would now average 1.21.

 

So it is indeed true that once PolitiFact "dissolved its partnership" with Congressional Quarterly, it began to appear as though it was increasingly assigning Falses as Pants on Fire to Republicans.  But working with CQ may not be a cause even with such a correlation, given that its own writers' assignment of Pants on Fire to Republicans were far more (58% more) than that of what PolitiFact National was doing at the end of 2008.  Were there other possible reasons?  What happened in 2009?