Pants on Fire Factor: Obama-Fact

The addition of the state PolitiFacts began with the appearance of PolitiFact Florida in 2009, but really didn't really get in gear with publishing a substantive number of rulings until 2010.  In fact, in 2009, PolitiFact "Florida" published a grand total of 9 rulings, bringing in lead PolitiFact Florida writer Aaron Sharockman around October.  Then things took off in 2010 as Texas, Georgia, Rhode Island were added in the first half and then Ohio, Oregon, Wisconsin and Virginia in the second half.

 

It would seem as if the states came along to fill in the gap of the Congressional Quarterly, whose writers worked with PolitiFact doing rulings from 2007 through 2008.  However, as I described in my original assessment of this group and what its "affects" may have been , the types of statements selected at that time were of a different nature:  in the case of Congressional Quarterly, they related to its area of expertise, congressional positions and votes (although they did do other types).  In the case of the PolitiFact states, however, many rulings had to do with matters of a local nature, for example, one of the first PolitiFact Florida rulings on  a statement by then-Governor Charlie Crist who was bragging up the reduced rates of a negotiated, un-subsidized Florida healthcare program called Cover Florida Healthcare (found False).

 

The "Pants on Fire factor" (PoF) as I call it, a measure of what is claimed to be PolitiFact's bias for rating a False ruling as Pants on Fire, is shown below as aggregated from 2007 to September of 2012.    It is a comparison of Democrats to Republicans, with a factor of 1.0 translating to mean that PolitiFact assigns Pants on Fire to Democrats at the same (or equal) rate it assigns Republicans.  Anything below the 1.0 means Republicans are assigned Pants on Fire less than Democrats, anything above the 1.0 means Republicans are assigned Pants on Fire more than the Democrats (as a percentage of False + Pants on Fire rulings).

 

 

The red line shows the growth of the factor from favoring the Republicans to favoring the Democrats (or growing ever more negative for Republicans) over PolitiFact National's six years of Truth-o-Meter rulings.  The chart begins in quarterly increments until the end of 2008 to show how the Congressional Quarterly scored for that period as well, displayed in the green line. Thereafter the increments are annual.

 

At the end of 2008, PolitiFact National (including the Congressional Quarterly) had a factor below 1.0, although as shown by the green line, the Congressional Quarterly (CQ) alone was assigning Pants on Fire to Republicans 42% more than to Democrats.  With CQ behind them, in 2009 the PoF factor made its biggest jump ever at PolitiFact National, from .90 to 1.60.    For the year 2009 alone, PolitiFact National's Pants on Fire factor was a whopping 5.06.

 

Accumulated through the end of September of 2012, it now stands at 1.94--in other words, in the course of its history, PolitiFact National assigned Pants on Fire almost twice as often for Republicans as it did for Democrats.

 

Combined with the state PolitiFacts, however, the chart takes a different slope, as shown by the purple line.  It starts at the same high point as in 2009 because of there being only 9 PolitiFact Florida rulings, but as they accumulated, the aggregate Pants on Fire factor for PolitiFact as a whole dropped, from the 1.60 in 2009 down to its current 1.33.  The states themselves, if they were separated from PolitiFact National, would now average 1.21.

 

So it is indeed true that once PolitiFact "dissolved its partnership" with Congressional Quarterly, it began to appear as though it was increasingly assigning Falses as Pants on Fire to Republicans.  But working with CQ may not be a cause even with such a correlation, given that its own writers' assignment of Pants on Fire to Republicans were far more (58% more) than that of what PolitiFact National was doing at the end of 2008.  Were there other possible reasons?  What happened in 2009?

Liberal Proof of PolitiFact (Liberal) Bias: A Summary Thus Far

A better standard is needed for me to say something at PolitiFact (PF) looks like "liberal" bias than for me to be a Republican and just not like what they say like many of its detractors.   Or to pick something out of selected rulings that I think is "bad journalism" when it's just my opinion.  Let's review what I've come across so far that I have assigned to "Liberal Proof of PolitiFact Liberal Bias" because I honestly have to admit there isn't much way around explaining it otherwise:

(1)   Yes, there is some inconsistency on the application of tax rates in the rulings, especially what exactly an "effective tax rate" should consist of, as noted in this recent post.   A ruling from PF Ohio and one from PF Rhode Island are particularly suspect for not being as thorough as they could have been and as a result appearing to  jack up the Truth-o-Meter for Democrats.

(2)  The PolitiFact National writers do seem to assign a lot more Pants on Fire to Republicans than their stateside counterparts (as well as produce rulings which make the Republicans look less truthful overall), with two exceptions.

(3) The Republican Party of Virginia was correct in one part of their over-the-top study of PolitiFact Virginia this past July:  it did appear to "dump" positive Republican rulings in the weekend Richmond Times Dispatch; however, they may be wrong assuming most people only read the news on weekdays.

(4) For the first three months of 2012, a weekly trend analysis of the Democrat and Republican Truth Index  appeared to indicate that at the end of the month, ratings would "even up."  I haven't done any analysis since, but have it set up in my data to do it for the year 2012.  What this means--those with a conspiracy mindset might think this was a deliberate attempt by PolitiFact to periodically even up the Truth-o-Meter (or the Truth Index) between parties.  I'm not sure what it means, and it might not occur again.

(5) Pat Boone's advertisement about Obamacare's IPAB rationing care got doubled up in Pants on Fire from two relatively-leaning-left-based-on-Truth-Index writers, but when a Georgia congress person made a very similar statement calling the rationing "the power to deny care to seniors" it received a Half True from PF Georgia.

(6)  In my version of what was to become my conservative counterpart's Pants on Fire factor, a measure of how much "more" Republicans are assigned Pants on Fire rulings, I looked at the Pants on Fire for PolitiFact's first 5,000 rulings and then proportioned them to the percentage of rulings overall.  The Republicans had 2.5 times as many Pants on Fire as Democrats, not considering the percentages.

7)  I also looked at the Truth Index in terms of writers,  by occupation (officials versus PACs, etc.) and by what I judged to be "local" rulings, for the year 2011, with varying results, although for the most part, they would predominantly favor the Democrats.  The one exception I found for what I judged to be "local" rulings, done for the first time in the final quarter of 2011,  was totally reversed in the first quarter of 2012.

8)  PolitiFact's most egregious problem may be "cookie cutter" rulings, where rulings are duplicated and serve to amplify the Truth Index negatively,  although the claimant often goes without political affiliation even though it's obvious what that affiliation probably is.    A major one this year was the across-the-board claim by "bloggers" that all the Democrats in congress were card-carrying socialists,  where almost all the PF states with a Democrat Congress member accused of "carrying the card" decided to jump on board and basically copy/paste the same ruling, inflicting a lot of Pants on Fire on those bloggers who had to be Republicans.  It could also be a subject as seemingly innocuous as light bulbs, a popular one in mid-2011.    Now it's been turned on its head a bit with the doubled up-assignment of "True" to a Democrat. 

Now, there's a few rulings of the same genre where PolitiFact has not been kind to the Democrats, and an excellent case of what could be termed Republican bias by PolitiFact Wisconsin.  And it would not take a lot of analyses to refute a few of my findings.  It just needs to be neatly summarized and put out there.

 

 

Yes, Virginia, PolitiFact isn't a Bunch of Social Scientists

Still No Real Proof of PolitiFact Bias from PolitiFact Bias

Well, I guess all the relentless pressure worked (ha ha!). After six months of nothing on PolitiFact Bias’ Research page, we finally have that all-encompassing, definitive, proof-positive, documented study to end all studies, that gives us evidence beyond a shadow of a doubt, and even includes a measure, of PolitiFact’s liberal bias.

That's all just sarcasm, of course,  because after downloading and reading the 6-page report, it reminded me of another “study” which was posted last month, a fact-check by PolitiFact New Jersey, about a liberal blog called Uppity Wisconsin which had added up all the Trues and Mostly Trues of several governors and concluded that that since there were less for Republicans compared to Democrats, Republicans must lie more (which made it all the way to a claim using the data by a New Jersey state representative, which was then picked up by PolitiFact New Jersey).

But PolitiFact Bias’ methodology was a bit more sophisticated, and went the other way on the Truth-o-Meter. It added up all the Falses and Pants on Fire from PolitiFact’s beginning through the end of 2011, then took a percentage of the Pants on Fire of the False + Pants on Fire, then compared that percentage between Democrats and Republicans as a way to show how many more Pants on Fire are given to Republicans. It appears that PolitiFact Bias’ Bryan White may have gotten the idea from Eric Ostermeier’s much lauded study where he also took a look at a similar percentage (and probably Uppity Wisconsin did as well, in terms of True and Mostly True).

Overall results pointed to a marked liberal bias from PolitiFact’s start in 2007 through the end of 2011. The percentages for groups A and B varied only slightly. For group A, Republicans found making false statements received a “Pants on Fire” rating 28.57% of the time and Democrats 16.42 . For group B the percentages were 28.23% and 15.88%, respectively--a very close match. Group C reflected the expected distortion from unreliable chain email claims. Though neutral judgment ought to result in approximately equal proportions of unfair “Pants on Fire” ratings, Republicans drew that designation about 74% more often than Democrats. The numbers show a clear bias against Republicans and conservatives.

For further study and comparison I will use the percentage of one party’s “Pants on Fire” ratings divided by the percentage of the other party’s “Pants on Fire” ratings as the “PoF Bias” number. For Group A from 2007 through the end of 2011 Republicans suffered with a 1.74 PoF Bias number.

I’m not sure how he could place such confidence or reliance in such a number as being “a clear bias against Republicans” or how in the world could he expect that “neutral judgment ought to result in approximately equal proportions of unfair Pants on Fire ratings.”

To repeat PolitiFact Editor Bill Adair’s own words from the PolitiFact New Jersey fact-check:

PolitiFact Editor Bill Adair called suggestions that a governor is "in the top five" for falsehoods "inaccurate -- and meaningless.” ….

"It's fine if people want to use our report cards to compare the records of people we fact-check. But it's important to be accurate about what those report cards say and what they don't," Adair said. "The report cards provide a tally of the claims we chose to check. But it's not accurate to say the report cards indicate who ‘told the most lies.’ We are journalists who choose our fact-checks based on what is newsworthy. We are not social scientists and are not using any kind of random sample to select statements to check."

Adair re-iterated his “selection bias”—what is newsworthy—but according to him, tallying up the rulings to make comparisons of truthfulness is “inaccurate – and meaningless.” I would venture Adair would say the same thing even with tallying up claims at a macro level as done by White.

While I’ve taken a lot of guesses—it could be liberal bias, it could be Republicans lie more, it could be the rise of Fox News or the Tea Party, it could be a lot of things—but I don’t think just adding up the Pants on Fire as a percent of all claims found false validates any conclusion of liberal bias.   There's no  PolitiFact "Achilles' Heel" here as White wants his readers to believe.

So, since I had the same database, I decided to see if I could first come close to confirming White’s results….and I did, only my number was a bit higher at 1.79 (his was 1.74). This was for all PolitiFact National claims through the end of 2011. A lot of minor things could have caused the variance, in particular the classification of political affiliation for chain e-mails, many of which were False and Pants on Fire. White also delineated conservatives from Republicans, which could be a very subjective process. I never separated conservatives from Republicans just as I never separated liberals from Democrats. Nevertheless, it was very close.

But the PolitiFact (PF) states, one would surmise, should be pretty close to that of National; after all, they were likely “trained” in the process of fact-check by PF National, they rate national figures (Eric Cantor by PF Virginia and John Boehner by PF Ohio, for example), and PF Florida is basically a clone of PF National (read: Angie Drobnic Holan). They exceed PF National in number, comprising 53% of all rulings. Plus, we have the office holder distinction as used by Eric Ostermeier: it “adjusts” out the “expected distortion of chain email claims” as well as media pundits, advocacy groups, PACs and websites.

Next, I thought I should include the Truth Index for the identical data which created the “PoF bias” factor, which takes into account the other Truth-o-Meter categories, to see if there was any correlation. After all, if there’s more Falses and Pants on Fire, it should follow there are fewer Trues and Mostly Trues, since it’s basically a zero sum game. I did two forms of three scenarios, marked A, B and C.

Truth Index: Friday is Fact-Check Day!

In my last review of PolitiFact Detractors, I published a response to a very one-sided, gigantic opus of talking points put together by the Virginia Republican Party, about the various ways they thought PolitiFact (PF) Virginia was biased in the way it did its Truth-o-Meter rulings. One of their complaints was called the “Weekend Dump” where they wrote that “PolitiFact [was] More Likely To Dump Positive Republican Rulings Late On A Friday, Or Over The Course Of The Weekend, As Well As Save Negative Republican Rulings For Publication During The Week (Monday Through Thursday).” (Don’t Know Why They Have To Start Every Sentence With A Capital Letter But That's How I Copied It!)

All that was needed  was to add a new field for the day of the week in my data. I already had the publish date so it was a easy task. I found that yes, using PolitiFact's Truth Index as a measure, for PF Virginia, there were more positive rulings for Republicans on weekends, and they were more negative during the week, but it was essentially a similar theme for the Democrats, contrary to the anecdotal rulings the Virginia Republican Party pointed out, at least for the first six months of 2012.

Then I thought I’d take a look at PolitiFact National, because it occurred to me when the Virginia Republican Party said there were rulings "buried" in the Saturday Richmond Times-Dispatch, I thought they could be those of PolitiFact National, since very few rulings are ever published on Saturday (most are by PF Oregon).

It should be mentioned that when I did the final filter, I found that PolitiFact National did not publish rulings on Saturday or Sunday (at least thus far in 2012). So I had to assume everything “over the weekend” as what was published on Friday.

Click to enlarge:  Certainly looks like more negative Republican rulings during the week at PF National.

PolitiFact National did 191 rulings in the first half of 2012, 130 Republican and 61 Democrat. From Monday through Thursday it published 154 rulings, 105 on Republicans and 49 on Democrats, and the rest (37) on Friday (25 Republican and 12 Democrat). During the week the Democrats had rulings which averaged a positive Truth Index of 1.02, whereas the Republicans were much lower, at a Truth Index average NEGATIVE 47.6, with 17 Pants on Fire rulings (the Democrats had only 2). With the Friday rulings over the weekend, however, the Democrats dipped to a Truth Index of negative 8.3 and suddenly got very close to the Republicans Truth Index, which made a marked improvement from negative 47.6 to negative 10.

Politi-Score: More Red in Ohio

During the 2010 election cycle, a trend was noted concerning the PolitiFact (PF) State Truth-o-meter rulings: that the “redder” affiliated state newspapers (ones that, for example, endorsed more Republican candidates) tended toward rulings, when averaged out through the Politi-Score, to either be more balanced between the parties, or to even favor Republicans, especially when compared to PF National. So how did that work out now with ALL the rulings instead of two months’ worth?

Before starting, I should note that I will be changing my “About This Blog” page shortly to reflect I am going with what I call “Wide Measure” rulings only from now on. I do like the innateness of 100 points for True, 50 points for Half True and zero for False.

That being said, here are the Democrat and Republican Politi-Scores compared for all states, for “former and current office-holders” only, according to that measure.

The “50” for Half True is the first focus on this chart. The averages ranged below 50 (Half True), but only PolitiFacts National and Florida, who have some of the same writers, had scores above 50 for the Democrats. Since together they dominate over 3/4 of the rulings, it caused the “ALL” scoring in the last column to average to above 50, even though all other PF states scored 50 or under collectively (for both parties).

Politi-Score: "My Bad" Boehner

In my initial review of Eric Ostermeier’s article “Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats” I deduced that he had used PolitiFact National statistics only. As a matter of uniformity, this was probably the most “scientific” way to do it. But because his total number of 511 statements still seemed a little low, and he did not come out and say that this was PolitiFact National, I puzzled over how I could determine it.

Since Speaker of the House John Boehner is from Ohio, I thought I might manually confirm his rulings. Indeed, he had some rulings by PolitiFact National and some by PolitiFact Ohio, here’s how his count worked out for 2010 when he received 21 PolitiFact rulings:

Boehner 2010	PF National	PF Ohio	Total
True	6	4	10
Mostly True	2	0	2
Half True	1	2	3
Barely True	1	0	1
False	3	2	5
Pants on Fire	0	0	0
Total	13	8	21

So right there (in emphasis), as Ostermeier noted, six Trues, two Mostly Trues, and one Half True. Yes, Ostermeier had compiled Boehner’s numbers for PolitiFact National only.

The issue raised for me was the way I had gathered my data for my “2,788 Reasons” series of posts. I accomplished the task by going to the “Other People and Groups” page of the PolitiFact.com website, and linking to each person, and then copying the numbers from the “Statements by Ruling” list. But the issue was that the “Statements by Ruling” do not tell you which PolitiFact group the ruling was done by, and there are some overlaps. The other politico I checked was House Majority Leader Eric Cantor, since he is a representative from Virginia, and yes, he had one ruling done by PolitiFact Virginia.