2013: First Six Months' Fact-checking for the Big Three

PolitiFact has definitely "downsized" in the U.S. even with its recent international presence in Australia.  At least for the time being.

 

Since 2011, I've generally expected 475 to 500 rulings from PolitiFact for each quarter.  When 360 were produced for the first quarter of 2013, I attributed that to the writers taking a break after the 2012 elections.  But the second quarter was pretty much a mirror of the first, going from 361 to an almost equal number of 363 rulings.   The dropping of PolitiFact Tennessee and no rulings from PolitiFact New Hampshire would not be enough to make up the difference.  Basically PolitiFact has cut back 20 to 25 percent.

 

With fewer rulings to record, I thought I might try something different:  back in 2010 I took a look at FactCheck.Org in and of itself.  I've compared rulings on the same statements many times, and on an individual basis, I've looked at various politicos in terms of Politifact, Factcheck.Org (FCO) and the Washington Post FactChecker.

 

This time I decided to record all three of the most prominent political fact-checkers (PolitiFact, WaPo FactChecker and FactCheck.Org) for the first six months of 2013.  I also tried to make a comparison in terms of the Truth Index.  Between PolitiFact's Truth-o-Meter categories and the Gepetto Checkmark to Four Pinnochios (a "whopper") used at the Washington Post, it wasn't too difficult there.    Factcheck.Org (FCO), however, does not rate fact-checks  with Truth-o-Meters or Pinnochios like the other two.

 

Just like it did when I reviewed it in 2010, I found that FCO generally did not fact-check "truthful" statements, as the great majority of its findings are negative: either misleading or false (It appears the Washington Post FactChecker leans that way as well, although not nearly as much).   It was a very subjective exercise for me to attempt to "rate" them as well, so the "Truth Index" I obtained, in my opinion, should not be construed as entirely accurate.

 

For the most part I based my FCO "ruling" on the same criteria as PolitiFact.  "Misleading" statements which contained an element of truth "but ignore[d] facts which would give a different impression" were categorized as "Mostly False" while statements which were "inaccurate," or incorrect (or when the FCO article called the statement a "false claim," which occurred fairly often) were categorized as False.    There were no "Pants on Fire" or "Whoppers" for FCO.

 

The results I obtained can be shown in the chart below.

The two "smaller" fact-checkers tend to concentrate on facts likely false based on the "Truth Index."

Even with PolitiFact's downsizing, it is still the dominant fact-checker of the "Big Three" with over 3/4 of all rulings for the first six months of 2013.    As noted, it checks a lot more statements which, as can be seen are (1) "True" and, due to its state operations, (2) statements by people with no political affiliation.

PolitiFact had a 57 percent higher overall Truth Index than the Washington Post FactChecker, and it was 82 percent higher than Factcheck.Org.

Truth Index+: Buh-Bye, Bachmann

What can I say.

Michele Bachmann was someone I termed a statistical "outlier" when it came to the PolitiFact Truth Index+, because she was so far off everyone else in terms of it, as well as in terms of its category Pants on Fire. When she announced she would not run for her congressional seat in 2014, PolitiFact "honored" her by describing the reasons for such a feat:

She had a remarkable streak: Her first 13 ratings were False or Pants on Fire.

Bachmann went on to compete in the 2012 Republican primary for president. She earned her first True rating for a statement she made during a 2011 debate that then-Sen. Obama "refused to raise the debt ceiling because he said President Bush had failed in leadership."

When a question was raised about her veracity, Bachmann cited our work during one of the many Republican presidential debates. "After the debate that we had last week, PolitiFact came out and said that everything I said was true," she said at a debate in Sioux City, Iowa, on Dec. 15, 2011.

Alas, she was totally wrong. In fact, the week before we had rated her statement that Mitt Romney instituted "socialized medicine" in Massachusetts as Pants on Fire.

Bachmann has been extensively covered on this blog--in fact, my very last post--and has been subjected to the Politi-Score, that is, taking into account all the fact-checks done on her by every source. In her case, she had four (4) sources because of the Minnesota Public Radio Fact-Checker in her home state. And she required more work than the others I've measured using the Politi-Score, as I ended up with a 5-part series when I covered her in August of 2011. It is still worth checking out, as it goes over many of the statements she made which were checked multiple times: (at that time) of the 50 statements from the 4 fact-checkers, a third of them were duplicated, and two statements were checked by all four fact-checkers (and for the most part rated quite consistently, which says something about the integrity of the fact-checkers' processes).

 

61% of Bachmann's statements
were rated False, of which
42% were Pants on Fire

Her Truth Index+ alone, however, should be reviewed in proper context. PolitiFact provided her last ratings by percentage for 59 rulings (see chart at right), but that was all (the same they would do for anyone else). Here is how she looks (below) when you calculate Bachmann's Truth Index+ and compare it to the overall Truth Index+ (through 3/31/2013), as well as that for Republicans overall, for Democrats overall, and compared to her fellow Republicans in Congress, and as an added individual comparison, to the next best female Democrat in congress who probably gets closest to her in terms of making those occasional outrageous statements, Nancy Pelosi, although she's only been subjected to the Truth-o-Meter 22 times, I'm assuming because she relinquished her Speaker of the House position following the Republican mid-term election rout in early 2011(and didn't attempt a run for president as Bachmann did).

 

As can be seen, the Truth Index+ for all the other measures goes from 84% higher for the Democrats overall down to 54% higher for her fellow Republicans in the House of Representatives. I'd also venture to say that for those who've had 50 or more rulings, which puts her in a very special Truth Index+ category especially for females, she probably ranks dead last as far as the Truth Index+ (males AND females).  And the more rulings, the more tell-tale "for truthiness" is the Truth Index+.

 

Today, PolitiFact published an article re-iterating its "principles" with regard to doing rulings. Bill Adair, PolitiFact's creator and outgoing manager, stated under the heading "Choosing Claims to Check" that "we select the most newsworthy and significant ones" and in a bulleted list of "questions they ask themselves" in selecting a statement asked "Would a typical person hear or read the statement and wonder: Is that true?" And Bachmann certainly met that criteria in many of her statements.  

 

In some ways, hard-line conservativism and Tea Party associations made her a colorful character.  But I can't really say she will be missed. Well, we still have 19 months to go, and anything can happen...so I'll keep my mouth shut.

Sidebar: Official Errors

Currently (as I've posted about before) I am doing a very tedious, time consuming task with regard to past PolitiFact rulings (about 4,000 of them from 2007 through 2011):  I am refining my "rough" database to include specific dates, statement descriptions and claimant's (current) position.  When done, the database will be much more accurate than before with regard to these details, and hopefully will provide some greater insight into statistics regarding PolitiFact's Truth Index, as well as more reference material for general posts.

 

My biggest errors thus far have not been with the Truth-o-Meter rulings themselves, it has been with the "claimant position."  This was added when the scholarly Eric Ostermeier study of 500 rulings in early 2011 came out:  he had separated those in an official position with those who were not.  To some "not an elected official" might mean the party boosters, the media pundits, the organizations.  This was basically how I quickly classified them when I thought I'd try to replicate Ostermeier's results.  But there are many individuals who were not elected officials when they sought office, or were in a different office.  For example, before running for governor of Florida, Rick Scott was a businessman. Senator Marco Rubio was not in any official position when he ran for senator, although he had been prominent in state government.  The way these two should be recorded, up until they assume office, is not as an official.  In other words, a PolitiFact "rulee's" position should be recorded as the one they were in at the time they made the statement.   In the Scott/Rubio examples, I had them recorded (mistakenly) as officials up through 2010.  While most strictly Democrat-Republican Truth Index comparisons would hold, anything with regard to official/non-official would be suspect, because Scott and Rubio comprised a large number of rulings (as well as others).

 

Other than that, when complete, there should be new, very accurate chart data to report.   I am also considering a new statistically "relative" approach other than PolitiFact's own Truth Index and the Politi-Score, which I will be testing with the new "cleaned up" database.  In the mean time, thank you for your patience!

Truth Index: Obama-Romney One Month Out

Dreaming of a Michele anniversary gift

Recently I posted about Mitt Romney's 47% remarks, saying it might be the game changer and even though it was rated "True" by PolitiFact, it might be considered a candidate for Lie of the Year...unless, as I said "...Obama could make his own big gaffe" which would reverse Romney's course following that statement. Well did he ever. I got up Thursday morning hoping the first debate had just been a bad dream. But I did enjoy SNL's take on it, which was just about the whole show.

 

But life must go on and it's time again to look at the Truth Index trend line for Romney and Obama. It hasn't changed much (unlike the polls), so I thought I might add something to show what appears to be the "Truth Index trend" for presidential candidates during their campaigns, i.e., the Truth Index of 2008 Republican presidential candidate John McCain. We see how all three candidates had slipping Truth Indexes as their campaigns went on. In my last post on this, and as can be seen in the chart below, Romney had a higher Truth Index than Obama until after he announced his candidacy in 2011, after which it deteriorated substantially.

Click to enlarge: All 3 candidates' Truth Index fell off during the campaign period

This chart represents 427 rulings for Obama (which may explain why his Truth Index has not fallen as much in 2012), 183 for Romney, and 169 for McCain, from about June of 2007 to present. Today Obama's Truth Index has bumped down a bit to a positive 11.48, and Romney is at a negative 16.12. John McCain's Truth Index, most of his rulings from the 2007-8 period, is a negative 8.88, roughly half that of Romney based on a similar quantity of rulings. For what it's worth, looking at the infamous "Pants on Fire" factor on an individual basis, Pants on Fire were assigned nearly three times as much to Romney than Obama (PoF 3.86), and twice as much to McCain than Obama (PoF 1.97).

Truth Index: CQ Tips

When PolitiFact first began its operations in 2007, it was a "joint venture" with the Congressional Quarterly (CQ), which until 2009, was owned by the same company as the Tampa Bay Times. This may be the reason why, at the end of 2008, the partnership with CQ was dissolved.

 

A claim by PolitiFact Bias in its "research" necessitated my review and update of 2007-10 statistics collected (which I had not dated or listed the writer) to determine what the CQ "input" might be:

When PolitiFact first launched, it existed as a joint venture between the St. Petersburg Times newspaper and the Congressional Quarterly magazine, both owned at the time by the Poynter Institute in St. Petersburg, Florida. The numbers for 2007 and to a lesser extent for 2008 demonstrate tendencies not present in later years. Some of the differences may stem from the vigorous Democratic primary season or from the removal of input from CQ staff when the partnership was dissolved after the 2008 presidential election. Simply put, PolitiFact’s bias against Republicans as measured by its use of the “Pants on Fire” rating has intensified since the separation from CQ. Republicans during the more recent era receive a “Pants on Fire” rating 143% more often than Democrats (see Fig. 2).

At this point I have reviewed enough data to have completely recorded all rulings for 2007-08, the period of time that CQ participated. CQ has a large staff and spread out PolitiFact rulings among many of their writers and researchers, but because the Congressional Quarterly itself was considered the primary publication for the CQ staff, it would appear that there were a only a few people assigned to PolitiFact full time. That being said, during the time they worked with PolitiFact, they published 170 rulings out of about 800 total, about 20 percent.

 

A point needs to be made with regard to the rulings in that, because the Congressional Quarterly's area of expertise seemed to be what was happening in Congress, the majority of its rulings appear to relate to that as shown in the graphic below on Joe Biden: votes and positions in Congress. This may serve to also assure that "Pants on Fire" is not as much an option.

Click to enlarge:  These are all rulings authored by CQ staff members--"bills".

So let's take a look at the period from the third quarter of 2007, about when PolitiFact began to get established and enough rulings accumulated, and compare its Truth Index without the Congressional Quarterly rulings to that of just the Congressional Quarterly rulings. What we might expect is more "neutrality" to show--perhaps even a Truth Index favoring the Republicans, or at least very even, along with the crucial Pants on Fire indicator designed by PolitiFact Bias, to be around 1.0, or I'd say at least 5 percentage points within it, meaning a fair assignment of Pants on Fire. As I finish the dating for 2009 and 2010 data (I am 80% complete at this point) a chart can be built to see just how the Pants on Fire have "intensified" on a timeline.

 

The accumulated timeline for Democrats and Republicans is shown below, with the CQ Truth Index showing in the dashed lines. In the beginning, it seems we get a hint of PolitiFact's so-called liberal bias, with a 44 Democrat Truth Index, although the number of rulings is not significant enough--CQ at this point had only contributed 21. By the end of 2007, however, the Truth Indexes converged a bit, with the CQ Republican Index in the lead for "truthiness."

 

By the second quarter of 2008 the CQ Democratic Truth Index surpassed that of the Republicans, ending at a positive 9.1 for the Democrats and a positive 4.6 for the Republicans. The PolitiFact-team -only rulings, however, showed the Democrat-favorable variance it shows to this day at negative 11.9 for the Republicans and positive 13.6 Democrats. For all PolitiFact rulings accumulated through the first half of 2012, this compares today to a near-zero +.3 overall for the Democrats and a negative 23.5 for the Republicans. In other words, while the Truth Index still favors the Democrats today, both parties have been eroding steadily since 2007.

Click to enlarge:  This is a graph showing the Truth Index as accumulated.

Chain e-mails.  The last column of the chart is titled "No chains" because of the large number of chain e-mails PolitiFact started investigating and reporting on. In this last column the Truth Index is adjusted to show what it would be without those rulings. Before I had separated the e-mails for affiliation based on a cursory review of the ruling; this time I made it much more than cursory to try to make the affiliation as accurate as possible.

Truth Index: Yes I know, Get A Life!

It was late in 2010 when I first thought about doing my own database of PolitiFact rulings.  So between Christmas and New Years of that year, when there was a brief stand-still in publishing due to the holidays, I went through PolitiFact's personality page, linking to each person or group on the list and then recording the count.  So each record reflected the name and political affiliation, along with a total by True, Mostly True, etc.

 

In 2011, I started a more detailed analysis.  Each record was one ruling:  I also included the date of the ruling, subject and writer.  Late in 2011, I decided to include a brief description of the ruling.

 

In doing so I realized an historical account of PolitiFact was needed--the "up to 2010" rulings should be recorded in the same way, as one by one rulings instead of by person/group.  Recently my conservative counterpart made allegations with regard to Congressional Quarterly (CQ), which was PolitiFact's partner in 2007 and 2008.  The only way I could test their theory--that CQ mitigated PolitiFact's alleged left wing bias--was to record those rulings in this way, so that a complete timeline could be created, and the CQ rulings could be filtered out for analysis.

 

Luckily, PolitiFact doesn't delete "old" rulings.  So I've been spending a lot of time in a somewhat tedious process, although it's been interesting reading some of the claims made back then, as well as seeing how PolitiFact grew and progressed as a fact-checker, which I will post about soon.

 

Between other things going on in September (personally) and doing this, I've not done a whole lot of posting.  It should be picking up soon, however, as we look at PolitiFact with an historical perspective.

 

Truth Index: Obama-Romney 6 Weeks To Go

And a Peak at the Congressional Quarterly

For reasons to follow in this post, I thought it might be a good idea in this update on Obama and Romney's running PolitiFact (PF) Truth Index, to get a better look at what happened before 2010. So I haven't been posting as much, because it meant I had to go through 272 rulings for Obama and 44 rulings for Romney to date all the rulings...and do a few other things. 

Click to enlarge:  Until Romney veered way right, he had an admirable Truth Index

This latest chart shows the accumulated Truth Index for Obama and Romney quarter to quarter starting at the end of the third quarter of 2007, when the PolitiFact operation was really gearing up, since at that time as Obama fought for the Democratic presidential nomination, Romney also tried unsuccessfully to do likewise as a Republican. A note on the chart (brackets indicating the flat trend line) indicates the result at PolitiFact once John McCain became the presumptive 2008 Republican presidential nominee: from the end of the first quarter of 2008 until early in 2010, Romney was not subjected to the Truth-o-Meter at all. In early 2010, he was interviewed as the Affordable Care Act was being signed into law by Obama (perhaps because of his "Massachusetts" knowledge), which instigated a few rulings, and here it was that Romney perhaps hinted at one his future campaign talking points when he stated that he thought Obama was too soft (full of "apologies") in his foreign policy. He disappeared from PolitiFact's prying Truth-o-Meter until early 2011, and from then on he was a fairly regular PolitiFact fact-check target until his nomination put the fact-checks on (maybe temporary) steroids. He now has 174 rulings, which I believe is the second highest for PolitiFact.

 

But the trend shows, just as my last post with monthly trends showed, that Romney's Truth Index fell off considerably once he announced his candidacy for the second time in 2011. Of seven Falses awarded Romney between April and September, six of them were bad enough to be made Pants on Fire. He had only one True ruling during that time. By 2012, his average leveled off somewhat around negative 15.

 

Obama's Truth Index has been pretty steady in the positive 12 area (with 416 rulings as of this post), although the pre-2010 trend indicates he got off to a rocky start. He started with a very respectable 30 and then plummeted in the very next quarter to a negative 11. His Truth Index for that quarter only was a negative 62.5, with four Falses, with not one True nor one Mostly True. Must have been that the PF honeymoon with Obama was over.

 

The CQ Connection: A theory set forth in a PolitiFact Pants on Fire analysis by my conservative counterpart was that the Congressional Quarterly had influenced a less biased approach to the rulings, that PolitiFact had gone wild with Pants on Fire once the CQ partnership "was dissolved" at the end of 2008:

When PolitiFact first launched, it existed as a joint venture between the St. Petersburg Times newspaper and the Congressional Quarterly magazine, both owned at the time by the Poynter Institute in St. Petersburg, Florida. The numbers for 2007 and to a lesser extent for 2008 demonstrate tendencies not present in later years. Some of the differences may stem from the vigorous Democratic primary season or from the removal of input from CQ staff when the partnership was dissolved after the 2008 presidential election. Simply put, PolitiFact’s bias against Republicans as measured by its use of the “Pants on Fire” rating has intensified since the separation from CQ. Republicans during the more recent era receive a “Pants on Fire” rating 143% more often than Democrats (see Fig. 2).

Part of my effort is not just in dating the PolitiFact rulings, but determining which ones were written by a Congressional Quarterly writer. My interest was to see how the writers treated Democrats and Republicans in terms of the Truth Index, to see if there's any correlation to support PF Bias writer Bryan White's claims. But to do so is an enormously tedious exercise, because PolitiFact National has roughly 1,780 rulings to go through for almost a four year period.

 

So to start I did Obama and Romney (for purposes of this post), with John McCain in my sites to next do a "presidential candidate trend" comparison; but at this point I've only done less than 20 percent of the rulings, so it will be a while. That being said, a few points about the Congressional Quarterly (CQ): between Obama and Romney CQ writers wrote about 15 percent of the rulings: 39 for Obama and 9 for Romney. The Truth Index as computed for only the Congressional Quarterly writers was within half a point: a positive 5.1 for Obama and a positive 5.6 for Romney. The Pants on Fires seem to defy White's contentions, however: Romney received only one from a CQ writer, while the three Pants on Fires Obama received were all from PolitiFact writers.

Truth Index: Romney and Obama Update

Through the end of August, with about two months plus one week left until the election, Obama and Romney's accrued Truth Index scores have remained about the same since my last report.

Click to enlarge: Romney was "truthier" than Obama until he announced his candidacy for president.

Both Romney and Obama have experienced a slight uptick to their Truth Indexes as the election campaigns rev up; Romney's has stabilized, and he's actually been awarded quite a few "Trues" lately, with the exception of one Pants on Fire, a repeat of his "Obama Worldwide Apology Tour" claim.

Obama just surpassed his 400th ruling (the chart is based on 403 rulings as of August 31); he has been the "most ruled on" person at PolitiFact for obvious reasons.  Mitt Romney is far behind with 162 rulings and he has a few more to go before he surpasses John McCain (at 169) which will then make him Number Two.  That should happen very soon. 

My plan is to update this Truth Index chart again at the end of September, although my feeling is it will look almost the same. 

Oh yes, and lest I forget, PolitiFact is almost four times as likely to make a False "Pants on Fire" for Romney as it is for Obama, according to a certain PolitiFact detractor's dubious measure. 

Truth Index: The *Sweet* Sixteen

PolitiFact’s adding the percentages to the Truth-o-Meter tally on its website made me realize that when it came to individual rankings, I was wasting my time with my spreadsheets. I could do the individual tally “real time” as long as I knew who had the most rulings…which for the most part isn’t too hard to figure out.

Those who ran and who are running for president, or are an official in a PolitiFact state, got the most rulings. John McCain, Hillary Clinton and Sarah Palin are hardly ever subjected to the Truth-o-Meter these days, but they’ve had a bigger share of PolitiFact rulings overall than a lot of people making the news today because of the 2008 election campaigns. I suspect by election day,the newly selected Republican vice presidential candidate Paul Ryan might get up there in number to make the list, especially with PolitiFact Wisconsin’s help, although he has a way to go.

There are sixteen individuals who’ve had at least 40 PolitiFact rulings, and they comprise a bit over 25% of all of them. Here they are in order of the highest Truth Index to the lowest, and congratulations Hillary—appearing more “truthful” doesn’t seem to correlate to “winning” or getting one more votes, does it?

They are more of the representative sample that should be used in any “testing” of the Truth-o-Meter as to “truthfulness” or that supposed bias, but I won’t get into that here.

For those interested, for the top 16, the Democrats comprised 35% of the rulings (yes, for Hillary, Obama and Joe Biden!). The Democrat Truth Index was a positive 15.5, compared with an almost zero (.22) for all Democrat rulings; the Republicans a negative 16.8, compared with a negative 23 overall. (AND for what it's worth, the percent of Pants on Fire measure was 1.92 in favor of the Democrats.)

Truth Index: Romney and Obama--90 Days To Go

As of August 1, 2012, Mitt Romney and Barack Obama had 535 Truth-o-Meter rulings between them, which should make for an interesting measure when compared, since together they make up roughly nine percent of all PolitiFact rulings since its inception.

In May I posted the month-to-month accumulated index for Obama and Romney (he was Politi-Scored in April) and noted that he started out at the end of 2010 with a higher Truth Index than Obama, which is an achievement since Obama’s Truth Index has almost always been relatively high and in positive territory. But following PolitiFact’s rating of statements made in his presidential announcement and then what amounted to quite a few not-so-truthful criticisms of Obama as his campaign went on, he has dropped almost 35 points, from a positive 19 to a negative 16 overall. This chart shows the accumulated average since December, 2010.  It is the basically the chart posted in May extended for four months.

Click to enlarge: Romney's anti-Obama campaign statements are what has brought his Truth Index
down the most.

As noted on the chart, the big drop on Romney’s Truth Index appears to have occurred in June, 2011. And, wouldn’t you know, June 2 was the date Mitt Romney announced he was running for president, at a farm in Stratham, New Hampshire. The big dive for Mitt came in the second quarter of 2011 as shown in this chart, which is the non-accumulated Truth Index; that is, for example, in that second quarter, Romney only had 5 rulings which together measured a horrific negative 110 Truth Index.

Click to enlarge:  Romney bounced back after all his presidential announcement Pants on Fires, but
not to where he was before, when he was "truthier" than Obama.

In reviewing the quarter, we find that Romney started off with a ruling of Pants on Fire on May 13, where he had repeated PolitiFact’s 2010 Lie of the Year, that President Obama’s health care law "represents a government takeover of health care.” Yet in that ruling it was said of Romney, in the equivocal way which PolitiFact often speaks of the Truth Index:

Mitt Romney has a relatively even PolitiFact record, with roughly the same number of True and Mostly True statements (35) as Mostly False, False and Pants on Fire (34)

On June 2, Romney made his presidential announcement speech, where PolitiFact took him to task for saying "we are only inches away from ceasing to be a free market economy" and that Obama “traveled around the globe to apologize for America” which garnered two more Pants on Fire.

This was followed by a statement on June 16 that “this is the slowest job recovery since Hoover” found False by PolitiFact, and the fifth and last statement rated for the second quarter of 2012, found Half True, that "Obamacare takes $500 billion out of Medicare and funds Obamacare” which was probably deemed Half True because of the way in which he said it (unlike one of the candidates running against him).

For what it’s worth (in the interest of certain anti-PolitiFact blogs), I thought it might be entertaining to look at the “Pants on Fire” factor, a relative measurement of Pants on Fire awarded as a percent of False, under the defective assumption that it proves liberal bias on the part of PolitiFact. And yes, it appears that Romney received the rating Pants on Fire about 2.75 times more than Obama did, for all the False and Pants on Fire rulings between them. That compares to the 1.74 as computed for all of PolitiFact National.

Yes, Virginia, PolitiFact isn't a Bunch of Social Scientists

Still No Real Proof of PolitiFact Bias from PolitiFact Bias

Well, I guess all the relentless pressure worked (ha ha!). After six months of nothing on PolitiFact Bias’ Research page, we finally have that all-encompassing, definitive, proof-positive, documented study to end all studies, that gives us evidence beyond a shadow of a doubt, and even includes a measure, of PolitiFact’s liberal bias.

That's all just sarcasm, of course,  because after downloading and reading the 6-page report, it reminded me of another “study” which was posted last month, a fact-check by PolitiFact New Jersey, about a liberal blog called Uppity Wisconsin which had added up all the Trues and Mostly Trues of several governors and concluded that that since there were less for Republicans compared to Democrats, Republicans must lie more (which made it all the way to a claim using the data by a New Jersey state representative, which was then picked up by PolitiFact New Jersey).

But PolitiFact Bias’ methodology was a bit more sophisticated, and went the other way on the Truth-o-Meter. It added up all the Falses and Pants on Fire from PolitiFact’s beginning through the end of 2011, then took a percentage of the Pants on Fire of the False + Pants on Fire, then compared that percentage between Democrats and Republicans as a way to show how many more Pants on Fire are given to Republicans. It appears that PolitiFact Bias’ Bryan White may have gotten the idea from Eric Ostermeier’s much lauded study where he also took a look at a similar percentage (and probably Uppity Wisconsin did as well, in terms of True and Mostly True).

Overall results pointed to a marked liberal bias from PolitiFact’s start in 2007 through the end of 2011. The percentages for groups A and B varied only slightly. For group A, Republicans found making false statements received a “Pants on Fire” rating 28.57% of the time and Democrats 16.42 . For group B the percentages were 28.23% and 15.88%, respectively--a very close match. Group C reflected the expected distortion from unreliable chain email claims. Though neutral judgment ought to result in approximately equal proportions of unfair “Pants on Fire” ratings, Republicans drew that designation about 74% more often than Democrats. The numbers show a clear bias against Republicans and conservatives.

For further study and comparison I will use the percentage of one party’s “Pants on Fire” ratings divided by the percentage of the other party’s “Pants on Fire” ratings as the “PoF Bias” number. For Group A from 2007 through the end of 2011 Republicans suffered with a 1.74 PoF Bias number.

I’m not sure how he could place such confidence or reliance in such a number as being “a clear bias against Republicans” or how in the world could he expect that “neutral judgment ought to result in approximately equal proportions of unfair Pants on Fire ratings.”

To repeat PolitiFact Editor Bill Adair’s own words from the PolitiFact New Jersey fact-check:

PolitiFact Editor Bill Adair called suggestions that a governor is "in the top five" for falsehoods "inaccurate -- and meaningless.” ….

"It's fine if people want to use our report cards to compare the records of people we fact-check. But it's important to be accurate about what those report cards say and what they don't," Adair said. "The report cards provide a tally of the claims we chose to check. But it's not accurate to say the report cards indicate who ‘told the most lies.’ We are journalists who choose our fact-checks based on what is newsworthy. We are not social scientists and are not using any kind of random sample to select statements to check."

Adair re-iterated his “selection bias”—what is newsworthy—but according to him, tallying up the rulings to make comparisons of truthfulness is “inaccurate – and meaningless.” I would venture Adair would say the same thing even with tallying up claims at a macro level as done by White.

While I’ve taken a lot of guesses—it could be liberal bias, it could be Republicans lie more, it could be the rise of Fox News or the Tea Party, it could be a lot of things—but I don’t think just adding up the Pants on Fire as a percent of all claims found false validates any conclusion of liberal bias.   There's no  PolitiFact "Achilles' Heel" here as White wants his readers to believe.

So, since I had the same database, I decided to see if I could first come close to confirming White’s results….and I did, only my number was a bit higher at 1.79 (his was 1.74). This was for all PolitiFact National claims through the end of 2011. A lot of minor things could have caused the variance, in particular the classification of political affiliation for chain e-mails, many of which were False and Pants on Fire. White also delineated conservatives from Republicans, which could be a very subjective process. I never separated conservatives from Republicans just as I never separated liberals from Democrats. Nevertheless, it was very close.

But the PolitiFact (PF) states, one would surmise, should be pretty close to that of National; after all, they were likely “trained” in the process of fact-check by PF National, they rate national figures (Eric Cantor by PF Virginia and John Boehner by PF Ohio, for example), and PF Florida is basically a clone of PF National (read: Angie Drobnic Holan). They exceed PF National in number, comprising 53% of all rulings. Plus, we have the office holder distinction as used by Eric Ostermeier: it “adjusts” out the “expected distortion of chain email claims” as well as media pundits, advocacy groups, PACs and websites.

Next, I thought I should include the Truth Index for the identical data which created the “PoF bias” factor, which takes into account the other Truth-o-Meter categories, to see if there was any correlation. After all, if there’s more Falses and Pants on Fire, it should follow there are fewer Trues and Mostly Trues, since it’s basically a zero sum game. I did two forms of three scenarios, marked A, B and C.

Truth Index by Subject since 2011

In a previous post on the subject of guns, I had written a huge paragraph on the overlap of subject assignment, which I ended up taking out when I realized that all I have to do at the PolitiFact website is click the label link for the subject to get all the rulings. But it’s still easier to do it on a spreadsheet because it’s all on one page right in front of me.

Anyway, the top ten subjects by order of number of occurrences (i.e., because of that ubiquitous selection bias) are shown in the chart below: They constitute roughly one half (48%) of all the rulings for which I kept track of the subject, for all of 2011 and the first half of 2012, 1,405 rulings out of 2,923 total. The chart shows the subjects highest count to lowest (starting with “Economy” with 234 rulings, working down to “Environment” with 67 rulings). Then it shows the Truth Index for Democrats and Republicans for each subject with the Truth Index score labeling each bar.

Click to enlarge: On the subject "Federal Budget" Republicans calculated to a "zero" Truth Index.

The chart does not show the total breakdown or average for these top ten: of the 1,405 rulings, 843 (60%) were from Republicans and 488 (about 35%) were from Democrats, with the remaining 5% Independent, Non-Partisan or unknown. The Republicans had an overall Truth Index for these top ten subjects of negative 21.5 (compared to negative 24.1 for all 2,935 rulings), while the Democrats had a positive 4.4 for these top ten (compared to a negative 1 for all 2,935 rulings).

The conclusions: while the economy was the most covered subject, in terms of number of rulings and Truth Index divergence between Democrats and Republicans, healthcare was by far the worst and most lied-about subject, especially by Republicans. For the year 2011, healthcare was also Number 1, and I’m sure this continued trend has something to do with their keeping the public turned against that dreaded “Obamacare.”

You can also check my recent post of some of the more repeated statements on the topic of healthcare, as well as PolitiFact rulings for which FactCheck.Org reached the same conclusion.

The Republicans were ahead Truth-Index-wise of the Democrats two subjects, jobs and “candidate bio”. Job creation has been lackluster under Obama, and so there may be a little stretching of the truth on the topic of jobs on the part of Obama and other Democrats to make it seem better than it is. As the political campaigns for 2012 start to take shape, “candidate bio”—something said about the candidate's opponent, his background or history—has been getting more attention for fact-checking verificiation, and here it seems the Republicans are a wee bit more “truthier” than Democrats.

The largest divergence between Truth Indexes without considering count, however, was on the topic of energy. I filtered out three “underlying reasons” just looking at the statements which I labeled Drill (Baby Drill), Gas (prices), and Keystone (Pipeline). When I looked at the “affiliation” list after executing the filter for these three items, I didn’t have to do anything more: there were 25, and they were ALL Republican statements, with a combined Truth Index of very negative 66. One of my favorites was a Pants on Fire for a Florida senatorial candidate by the name of Dave Weldon who said that there was no wildlife in the Arctic National Wildlife Refuge—sort of sounds like someone saying Ulysses S. Grant isn’t buried in Grants Tomb. Or a False from PolitiFact Ohio for a statement by Representative Mike Dovilla that Obama’s policy on the Keystone Pipeline was thwarting the ability to (immediately!) reduce gas prices.

Other than that, my readers can reach their own conclusions.

Truth Index: Don't Rush Me

After publishing the first half PolitiFact (PF) ruling results, it’s come time for more breakdown of the detail, and as I accomplished that I came across an “amusing” error I had made: I accidentally classified Rush Limbaugh as a Democrat! Oh woes me! So that’s been fixed. He only had one Pants on Fire ruling for the first half of 2012, but it did slightly affect the Truth Index.

Here’s how it breaks down by “source”—starting with the President/VP, the congress, the governors, the states and local governments, as well as advocacy groups, PACs and the media, and includes the overall Truth Index (here's the one for first quarter of 2012). “Others” includes former officials, blogs, chain e-mails and websites. A new category has been added called “Candidates (other than President)” as we have more and more of them due to the beginning of the 2012 election campaign. All officials I’ve categorized are “sitting”: in the case of presidential (I’d say ex-presidential but I don’t want to incur the wrath of Ron Paul fans) candidates like Ron Paul and Rick Perry, they are included in the categories of their “sitting” positions, congressman and governor.

Click to enlarge:  Groups (PACs etc) have a far lower Truth Index (-43) than individuals (-12) .

The following is a further breakdown of the pie chart, showing Democrats and Republicans. As can be seen the “local” category is now almost exactly even between Democrats and Republicans, whereas for the first quarter I showed a Republican advantage. The difference in the “local” Truth Index you’ve seen in earlier posts (as far as number of rulings) is that the chart indicates local politicians only, but there are also local issues covered by almost every source category which was what my earlier posts were about.

Click to enlarge:  FIRST HALF 2012: Only 3 rulings on Democrat governors made for the lowest Truth Index.

The senators and governors having more positive rulings than the other positions seems to be holding. The senators’ longer terms and their proverbial meaning of being “the eldest and wisest” (although let’s remember the word senator is from the same root as “senile” ha ha!) may have something to do with their being more “truthier” when subjected to the Truth-o-Meter. I posted something earlier on the predominantly Republican governors as far as who was the lowest on the Truth Index (Scott Walker of Wisconsin) but relatively speaking as a group they are much more “truthier” than their political counterparts.  

Next up, PolitiFact states or maybe PolitiFact writers?

Truth Index: Friday is Fact-Check Day!

In my last review of PolitiFact Detractors, I published a response to a very one-sided, gigantic opus of talking points put together by the Virginia Republican Party, about the various ways they thought PolitiFact (PF) Virginia was biased in the way it did its Truth-o-Meter rulings. One of their complaints was called the “Weekend Dump” where they wrote that “PolitiFact [was] More Likely To Dump Positive Republican Rulings Late On A Friday, Or Over The Course Of The Weekend, As Well As Save Negative Republican Rulings For Publication During The Week (Monday Through Thursday).” (Don’t Know Why They Have To Start Every Sentence With A Capital Letter But That's How I Copied It!)

All that was needed  was to add a new field for the day of the week in my data. I already had the publish date so it was a easy task. I found that yes, using PolitiFact's Truth Index as a measure, for PF Virginia, there were more positive rulings for Republicans on weekends, and they were more negative during the week, but it was essentially a similar theme for the Democrats, contrary to the anecdotal rulings the Virginia Republican Party pointed out, at least for the first six months of 2012.

Then I thought I’d take a look at PolitiFact National, because it occurred to me when the Virginia Republican Party said there were rulings "buried" in the Saturday Richmond Times-Dispatch, I thought they could be those of PolitiFact National, since very few rulings are ever published on Saturday (most are by PF Oregon).

It should be mentioned that when I did the final filter, I found that PolitiFact National did not publish rulings on Saturday or Sunday (at least thus far in 2012). So I had to assume everything “over the weekend” as what was published on Friday.

Click to enlarge:  Certainly looks like more negative Republican rulings during the week at PF National.

PolitiFact National did 191 rulings in the first half of 2012, 130 Republican and 61 Democrat. From Monday through Thursday it published 154 rulings, 105 on Republicans and 49 on Democrats, and the rest (37) on Friday (25 Republican and 12 Democrat). During the week the Democrats had rulings which averaged a positive Truth Index of 1.02, whereas the Republicans were much lower, at a Truth Index average NEGATIVE 47.6, with 17 Pants on Fire rulings (the Democrats had only 2). With the Friday rulings over the weekend, however, the Democrats dipped to a Truth Index of negative 8.3 and suddenly got very close to the Republicans Truth Index, which made a marked improvement from negative 47.6 to negative 10.

Grading PolitiFact *Liberal*-Style: Yes, Scott Walker is #1 "Liar"

A PolitiFact (PF) New Jersey fact-check on Loretta Weinberg, the state senate majority leader, done on June 7, seems to have slipped through my conservative counterpart’s bias-searching fingers, because I’m sure if he had read it he would be quoting it (besides having a major cow!) liberally throughout both his anti-PolitiFact blogs. For Bill Adair, the head of PolitiFact, came right out and directly addressed his biggest complaint of why he despises the Truth-o-Meter concept and what he believes is the reason for its alleged “bias.”

Weinberg said that Scott Walker of Wisconsin topped a list of the  five “Lie - en Governors" according to PolitiFact, and New Jersey Governor Chris Christie was also part of the “top five.” PolitiFact rated this statement “False” and included an interview with PolitiFact Manager Bill Adair, who made this very interesting comment:

"It's fine if people want to use our report cards to compare the records of people we fact-check. But it's important to be accurate about what those report cards say and what they don't," Adair said. "The report cards provide a tally of the claims we chose to check. But it's not accurate to say the report cards indicate who ‘told the most lies.’ We are journalists who choose our fact-checks based on what is newsworthy. We are not social scientists and are not using any kind of random sample to select statements to check."

But did Uppity Wisconsin, the source of Weinberg’s information, have it right? Here’s how PolitiFact explains what they did:

So what is Weinberg talking about?

The state senator’s statement was based on a list posted on the liberal blog Uppity Wisconsin.

The list ranked governors fact-checked at least five times by PolitiFact by the percentage of True and Mostly True ratings they received.

Walker had the lowest combined percentage of those two rulings, according to the June 1 blog post. The post concluded that "Walker is the #1 most dishonest governor in America."

Christie took fourth place among the "top five most dishonest governors in America" with 38 percent of his PolitiFact rulings earning True or Mostly True.

Uppity Wisconsin simply added the percentages of Trues and Mostly Trues, and came up with Scott Walker of Wisconsin at Number One with a combined percentage of 22 percent, followed by Rick Perry of Texas and Lincoln Chafee of Rhode Island both at 25%, Chris Christie of New Jersey at 38%, and lastly Bob McDonnell of Virginia at 44%. Uppity Wisconsin ignored a lot of things in coming up with these numbers. Adair has a point that this does not accurately show “who told the most lies.” For example, in 2012, Rhode Island governor Lincoln Chaffee had only two rulings from PolitiFact. He’s only had eight fact-checks ever from PolitiFact. This is not a good measure of “truthiness” for someone with this few fact-checks.

The more fact-checks on someone, the more it should bolster the claim of measuring honesty. But it’s even hard to say what that “more” number is. I put together a list of those sitting governors with 20 or more fact-checks, and decided "the top five" should be those with 40 or more: governors Chris Christie (NJ), Rick Scott (FL), Rick Perry (TX), John Kasich (OH) and Scott Walker (WI). I threw out Governor Nathan Deal of Georgia (with 22) and Bob McDonnell of Virginia (with 23), simply because I wanted to measure those with the highest number of fact-checks who had been rated recently. Not a lot of politicians meet the “40” criteria, but it’s a better measure than basing it on fewer than 10 fact-checks.

The other problem with Uppity Wisconsin’s measure is it does not take into account the Half Trues, Mostly Falses, Falses and Pants on Fire categories in making its "honesty" measure. The Truth Index as calculated by PolitiFact places heavier weight on the Falses and Pants on Fire categories: the Pants on Fire is weighted 50% higher than a True, because (I’m guessing) it was felt emphasis should be put on the false statements.

Truth Index: 2012 First Half

The second quarter of 2012 saw a decrease in Republican rulings by PolitiFact with the wind-down of the Republican primaries and Mitt Romney emerging as the presidential candidate. There was a slight decrease in the number of rulings, with 500 rulings published in total compared to 516 in the first quarter of 2012. The Republicans dominated in the first quarter with over 60% but for the second quarter dipped to just over 54% while rulings on the Democrats went from 34.3% to almost 39%.

A two-week segmented line graph shows that the trend disparity between the Republicans and Democrats also narrowed following the primaries. While the Republicans were for the most part less “truthful” than the Democrats, they did not vacillate as much as the Democrats did, and it was more the case that more statements of Democrats were found less truthful while the Republicans stayed about the same. While it was speculated after the first quarter that PolitiFact was doing “catch up” at month-end to even out party ruling outcomes, there does not appear to be any evidence of it now.

Click to enlarge:  the gap grew closer in the second quarter as the Republican primaries drew to a close.

One of the reasons there are more Republican statements being fact-checked is that they are simply more newsworthy Republicans in the states PolitiFact represents: Republicans dominate governorships and legislatures in Georgia, Florida, Ohio, Texas, Virginia and Wisconsin, whose PolitiFact branches as a group publish much more in the way of rulings than New Hampshire, Oregon, Rhode Island, and New Jersey.

Here are the overall charts comparing the Republicans and Democrats for the first half of 2012 in the PolitiFact app format.  It supports the comments above that while the Republican Truth Index stayed about the same at negative 26.1 (for the first quarter they were at a negative 23.8) the Democrat Truth Index fell off from a positive 5.4 to a negative 2.4.   I will be delving into (at least one of) the reasons for that drop soon.

Click to enlarge: Almost equal in percent of Half Trues and Falses, tho the Repubs have more than double Pants on Fire.

Truth Index: Local Update

In the last quarter of 2011, in my quest to find unique statistical trends from the data mining I had done of PolitiFact’s Truth-o-Meter rulings, I had assigned a field called “Local” to each of the PolitiFact rulings. As I described at the time:

What is termed “local” deals with the nature of the PolitiFact (PF) ruling. Most often it would show in the title or subtitle. It means it just relates to something having to do with the state or locality, that it’s not national. It could be ideological, but it still pertains to a local issue. None of these rulings were from PolitiFact National. Intuitively they wouldn’t, but state PolitiFact rulings can be done for an office holder of that (PolitiFact) state, while the subject of the fact-check is of national scope. In other words, without reviewing each ruling, the state rulings can't all be counted as local…

…Republicans are more truthful than Democrats….by a pretty good margin. Republicans were at a precise “zero” on the Truth Index, while the Democrats were a negative 27. There were 173 “local” rulings which represented 38% of all the PF rulings in the fourth quarter of 2011; Democrats and Republicans were also equally represented, each having 74 rulings. There were 25 rulings for which the affiliation was independent, non-partisan or just unknown.

That quarter I had 174 rulings of a local nature, and I got a somewhat surprising result in the Truth Index by party: the Republicans, as noted in the quote above from my post, were ahead, way ahead of the Democrats on the Truth Index (as compared to the Democrats normally being higher, as I have shown when you look at all rulings accumulated). I wondered whether it just an anomaly for the quarter. Although I was excited at this seeming incongruence, and confounded to determine an explanation, I still needed more data in order to confirm that this trend continued.

Click to enlarge:  The Republicans local "Pants on Fire" quintupled from 2011 to 2012, lowering their overall Truth Index.

So I now have six months of data to look at, almost 400 rulings (more than twice what I had the first time) and it seems my initial findings may have been premature….or maybe the Truth Index is functioning the way it should. In the first quarter of 2012, the Democrats have reversed the trend from the last quarter of 2011, and the Republicans reversed as well. The Democrats went from a negative 27 in the 2011 fourth quarter to a positive 11 in the 2012 first quarter for an overall average of negative 5.6. The Republicans went from an exact zero to a negative 12 for an overall average of negative 6.8. In other words, the Democrats and Republicans are statistically tied for the Truth Index average at around a negative 7 based on six months of Truth-o-Meter rulings. Compare this to a negative 13 for all rulings (well, the first 5,000) or a negative 9 for individuals.

But this also serves to prove in a sense that PolitiFact isn’t doing anything deliberately—the Truth-o-Meter goes up and it goes down, and it has nothing to do with “selecting” those statements which would benefit either side.

One of the reasons the Republicans “took a fall” this quarter was that there was a steep increase for them in Pants on Fire rulings, from two in the last quarter of 2011, to ten thus far in 2012. The pantalones en fuego ranged from such things as a statement by New Jersey Governor Chris Christie that “23 percent of kids starting ninth grade in Newark will graduate in four years” to PolitiFact Tennesee’s ruling on state representative Frank Niceley on his claim regarding the Tennessee Wildlife Resources Agency bringing coyotes into the state "to try to keep the deer population down." They were spread out over almost every PolitiFact state and didn’t all occur at any particular time during the quarter.

But this does show, so far, that overall, politicians appear to be a bit more truthful at a local level. I will continue to keep the “local” factor in the rulings and monitor them to get a better picture.

Truth Index: Quantity has a Quality All its Own

In a Politi-Score/Truth Index review of Mitt Romney, it was graphically shown how his Truth Index has deteriorated since entering the 2012 presidential race, from 2011 forward. At the end of 2010, it was more than 50% higher than Obama’s (a positive 19 versus Obama’s 12). At that time Mitt had 45 rulings total, which put him in the “exclusive club” of those with the number of Truth-o-Meter rulings which might mean something statistically.

By the end of March, 2012, Mitt’s Truth Index had dropped to a negative 12.7, while Obama’s barely moved, from just over 12 to under. But there is another way to look at this: what if Romney had the same number of rulings as Obama at the end of 2010, but with the same Truth Index of a positive 19 he had at the time? How would it look now?

Mitt talks about the Obama economy, Obama-care, Obama recession, Obama-blah blah blah. So let’s give Mitt an Obama-adjusted Truth Index. Based on an 18.9 Truth Index with 272 rulings (Obama’s number)as opposed to 45, Mitt’s Truth Index would today be a positive 8.02. It would still have gone slowly down, but obviously much more smoothly. Point wise he would be very close to Obama at 11.54.  Here's the graph:

Click to enlarge:  Obama and Mitt might draw close to even in next 6 months to the election (on quantity-adjusted).

So there is a flattening effect in the averages caused by the large number of rulings, even with the index heavily weighted to the False side. When Mitt hit 100 rulings, each additional ruling affected the average less than 1 percent, in Obama’s case, it’s now less than a third of one percent.

In a way, this shows the poor sensitivity of the Index individually—one more reason it should not be considered a reliable measure of “truthiness.” It should also be noted that Romney fell below Obama’s Truth Index in January…this may in part be due to Romney’s veering way right as the primaries started and the campaign ramped up. I will have to review this later to see if Romney’s Truth Index improves.