2013: First Six Months' Fact-checking for the Big Three

PolitiFact has definitely "downsized" in the U.S. even with its recent international presence in Australia.  At least for the time being.

 

Since 2011, I've generally expected 475 to 500 rulings from PolitiFact for each quarter.  When 360 were produced for the first quarter of 2013, I attributed that to the writers taking a break after the 2012 elections.  But the second quarter was pretty much a mirror of the first, going from 361 to an almost equal number of 363 rulings.   The dropping of PolitiFact Tennessee and no rulings from PolitiFact New Hampshire would not be enough to make up the difference.  Basically PolitiFact has cut back 20 to 25 percent.

 

With fewer rulings to record, I thought I might try something different:  back in 2010 I took a look at FactCheck.Org in and of itself.  I've compared rulings on the same statements many times, and on an individual basis, I've looked at various politicos in terms of Politifact, Factcheck.Org (FCO) and the Washington Post FactChecker.

 

This time I decided to record all three of the most prominent political fact-checkers (PolitiFact, WaPo FactChecker and FactCheck.Org) for the first six months of 2013.  I also tried to make a comparison in terms of the Truth Index.  Between PolitiFact's Truth-o-Meter categories and the Gepetto Checkmark to Four Pinnochios (a "whopper") used at the Washington Post, it wasn't too difficult there.    Factcheck.Org (FCO), however, does not rate fact-checks  with Truth-o-Meters or Pinnochios like the other two.

 

Just like it did when I reviewed it in 2010, I found that FCO generally did not fact-check "truthful" statements, as the great majority of its findings are negative: either misleading or false (It appears the Washington Post FactChecker leans that way as well, although not nearly as much).   It was a very subjective exercise for me to attempt to "rate" them as well, so the "Truth Index" I obtained, in my opinion, should not be construed as entirely accurate.

 

For the most part I based my FCO "ruling" on the same criteria as PolitiFact.  "Misleading" statements which contained an element of truth "but ignore[d] facts which would give a different impression" were categorized as "Mostly False" while statements which were "inaccurate," or incorrect (or when the FCO article called the statement a "false claim," which occurred fairly often) were categorized as False.    There were no "Pants on Fire" or "Whoppers" for FCO.

 

The results I obtained can be shown in the chart below.

The two "smaller" fact-checkers tend to concentrate on facts likely false based on the "Truth Index."

Even with PolitiFact's downsizing, it is still the dominant fact-checker of the "Big Three" with over 3/4 of all rulings for the first six months of 2013.    As noted, it checks a lot more statements which, as can be seen are (1) "True" and, due to its state operations, (2) statements by people with no political affiliation.

PolitiFact had a 57 percent higher overall Truth Index than the Washington Post FactChecker, and it was 82 percent higher than Factcheck.Org.

PolitiFact: A Truth Index Video!

2012 in Review at PolitiFact (Continued)

A few items have been posted already about the year 2012 at PolitiFact (PF).  It was their biggest year because of the presidential election, with 1,961 rulings produced.    Here's how it looked for all Truth-o-Meter categories as a percentage, in 2012:

Click to enlarge

In terms of Republicans versus Democrats, Democrats had a 2012 Truth Index+ of 58.5 (-3.7 PF Truth Index) for 735 rulings, while the Republicans had a 49.8 (-25.6 PF Truth Index) for 1,078 rulings.  That means that for the statements fact-checked by PolitiFact, the Democrats' rulings averaged about 17.5 percent higher (in proportional terms of Truth Index+).

Click to enlarge:  Truth Index+ for party is in parentheses

We've looked at month-to-month already, and took at a better look at Pants on Fire.  Let's take a look at the top ten "rulees"--those who were fact-checked.  This may explain to some degree why more Republicans are checked than Democrats.  While Obama was the person most fact-checked, he was the only Democrat on the top ten list.  Outside of people, only chain e-mails had enough rulings to get to the top ten.  The PolitiFact states showed their contributive power, with Chris Christie (PolitiFact New Jersey), Rick Scott (PolitiFact Florida), and Scott Walker and Tommy Thompson (PolitiFact Wisconsin) making it to the list.  Other than that, there were the expected presidential candidates (Vice president Joe Biden was Number 13 with 14 fact-checks).  The top ten shown reflects about 25 percent of all rulings; it should also be noted that of the 760 individuals/organizations "politi-facted" in 2012, 500 of them only received a single ruling (also about 25 percent of the rulings).  This shows how PolitiFact concentrates its efforts.

Click to enlarge:  Chain e-mails have always had one of the lowest Truth Indexes (highlighted).

In terms of topic, there is some overlap, that is, PolitiFact assigns multiple subjects by the nature of the fact-check, so these aren't quite as straightforward.  Generally, this is the "first" subject assigned in the fact-check, although it can be a subjective selection.  Occasionally it may be changed if I feel it's not correct, but that is rare.  Nevertheless, here are the top ten subjects which cover about 57 percent of all the rulings.  There's actually 11 because Deficits and Abortion tied for tenth place with 56 rulings each.

Click to enlarge:  Healthcare had lowest Truth Index because of high number of False and
Pants on Fire Obamacare claims (highlighted)

As far as separating Democrats from Republicans in other categories, I may be doing that as the year progresses, but more likely in terms of all rulings as PolitiFact quickly approaches 7,000.  Hold that dial.

 

Truth Index+: Office Holder Breakdown

The final post for 2012 will be an office holder breakdown from all PolitiFact rulings through the eve of the 2012 election (November 5, 2012).  The majority (about 53%, or 3,450) of statements put to the Truth-o-Meter at PolitiFact for fact-checking are from those in office:  from the president to senators and representatives, all the way down to local elected officials, such as mayors and city council members.   Here is how the official "pie" is cut:

Click to enlarge

Office holders as a group have consistently held a significantly higher Truth Index (14.7% higher overall) than that of those who were not in office.  For example, they had 43% more statements rated True, and 28% more statements rated True and Mostly True.  In the "subjectively rated" Pants on Fire category, office holders received just 45 percent of the Pants on Fire ratings percentage of those who did not hold office (5.4% versus 12.1%).

The reasons for this difference may be that a large portion of those "not in office" were advocacy groups and PACs, those without a "name" only seeking to promote a cause, and would not lose their voice in losing an election.  Pundits and chain e-mails, mostly in a position of "playing to the choir" could risk making ridiculous statements.

For Democrats versus Republicans, there's no Truth Index comparison for president/vice president: for most of PolitiFact's existence, the office of president has been held by a Democrat.  George W. Bush was only rated once prior to the election of 2008; PolitiFact preferred to concentrate on candidates Obama (168 rulings) and McCain (155 rulings) in 2007 and 2008.  So the chart below does not show any office of president comparison.

Click to enlarge:  Closest party to party Truth Index+ average was governors.

The (raw) Pants on Fire factor, a measure of how much more often a party is assigned a Pants on Fire when a statement rated by PolitiFact is determined False, shows a slight advantage for the Republicans overall and specifically in the categories of both houses of congress, governors and local governments.  The red (Republican) and blue (Democrat) bars are all the same--they all equal a one, with the "fire" bar a measure of how much more that party is assigned the Pants on Fire when they get a False.

Since there was no comparison for president (as I explained above), I decided to compare Obama to Romney for this chart, although Romney, since he was not in office, was not included in the 3,450 rulings.  It does show, however, quite a substantial variance between the two:  Obama has kept a very low percentage (2 percent overall) of Pants on Fire.  It may be a reflection of his very careful speech or maybe something to do with PolitiFact. 

Quantifying the PolitiFact Quants

There's certainly a lot of people out there scoring PolitiFact rulings... I'm finding more and more of them every day.  Presented and scored in different ways, in studies, articles and blogs, I think it may be time to add them as a special section to my PolitiFact Detractors page, though "detractors" are not quite what they are.

While I hope I'm one of the better ones, I do need to make the case.  I have probably looked at these rulings in the largest variety of ways, and have done so on an ongoing basis for quite a while (since 2010, when I started this blog).  I've looked at the score in terms of writer, topic, local versus national, by individual compared with other fact-checkers, etc.   I'm looking for patterns.  For example, if Republicans lie more, do they lie about every subject?  (No, there are some subjects for which they are more truthful than Democrats)  Are lies more ideologically based? (Yes)  Do some PolitiFact writers appear to favor one party over the other based on the Truth Index? (Yes) Most of the other PolitiFact quants have looked at them in terms of just the presidential contenders or Democrat versus Republican overall.  Some qualify their rulings as far as sampling/selection bias while others take the rulings at face value.

Straight Politi-Quant websites.  The website Who's More Full of Sh*t (authored by J. Kalgreen) graphically rates the same....with an ongoing feed from PolitiFact.  It uses a score heavily weighted to the negative rulings.  It also gives an overall Democrat versus Republican rating.  While it's a pure rating website, its website set up is quite sophisticated.  As noted by Benjamin "Brash Equilibrium" Chabot-Hanowell, "I seriously wish I could code CSS and PHP as well as J. Kalgreen [WMFoS author] can."

While a great site, Who's More Full of Sh*t might not be linked by PolitiFact because its name might offend some.

Instead of saying who's more "truthier" or "honest" it goes in the opposite direction--scoring zero for True down to 200% for Pants on Fire.  So if my conservative counterpart decided to try to troll-comment the site (which he can't because no comments are enabled) he'd have to complain that PolitiFact's selection bias is the cause for Democrats being less full of sh*t, or for Republicans to be more full of sh*t.  Because, you see, both are still full of sh*t.

Malark-o-Meter is more of the nerdy Nate Silver variety of  meta/probability analysis and in that respect is like no other PolitiFact quantifier.  A newer website based on (I'm guessing) Joe Biden's outburst at the vice presidential debate, author Benjamin "Brash Equilibrium" Chabot-Hanowell is interested in mathematically determining a possible "bias" factor as well as the probability of how much each claimant is prone to lie. He includes the Washington Post Factchecker rulings in his analysis as well.

"Serious" Studies.  I've posted about and often referred often to Eric Ostermeier's 2010 study of 511 PolitiFact rulings  where he concluded with the infamous selection bias statement.

The Center for Media and Public Affairs did a study of both PolitiFact and Washington Post Factchecker rulings but did not really reach any other conclusion ("[took] no position on their accuracy") as to why Republicans score as less truthful than Democrats.

And of course, the website PolitiFact Bias did a research paper on PolitiFacts False/Pants on Fire rulings, coming up with a percentage factor that I often use in my blog.    There are others who have looked at the Pants on Fire, some in a humorous way, or a lot like PolitiFact Bias.

Website Articles.  One of the very first to quantify PolitiFact's Truth-o-Meter rulings was the pseudonymous "Cinematical" at the Daily Kos, back in 2009....and arrive at the conclusion as so many that would follow, that conservatives were less "honest" than liberals.  There's also been several scorings at the Daily Beast, as well as a nice quant review by the New York Times' FiveThirtyEight.com (Nate Silver's blog).

Blogs.  Anyone math-oriented knew the moment they saw the Truth-o-Meter that it begged to be measured.  I imagine there's a lot more than just those mentioned here. So (besides myself), here's three very good ones:

In 2011, I came across the blog Quibbling Potatoes authored by Steve, who had some great ideas for categorizations of the rulings that I now use for my data.  He also had a name for his score which he called...the Truth Index.  While I don't think PolitiFact took the name from Steve's site, one can certainly say Steve had a bit of clairvoyance.  It's obvious he put a lot of work into his charts and graphs and it would be nice to see him do it again.

Pandering Politicians is a blog (turned website like PolitiFact Bias) by someone who calls himself "PanderBear" and the Truth Index here is called the "Truth Quotient."  He has a lot of charts, especially time trendlines, and begs people not to even read chain e-mails because of their dismal "TQ."  And of course, in one of the comment sections we see the ubiquitous question from my conservative counterpart asking what he does about selection bias.  No answer from PanderBear.  At least as far as I could see.

Allen Allermann in a namesake blog entitled "Random Human Neural Firings" did a quantifying review about the same time as Steve called "Who do You Trust?"  He scored PolitiFact rulings in the opposite direction that many tend to do--he gave higher weight to the True and Mostly True instead of False and Pants on Fire.  He uses a "sports" analogy for what Brash Equilibrium terms the report cards of those rated by PolitiFact; he calls them "at bats."

So there we have, including my own, about a dozen attempts to quantify PolitiFact rulings.  Perhaps two qualifiers can be added: (1) None of these websites/studies/blogs are affiliated with or approved by PolitiFact (as far as I know) and (2) all numbers should be interpreted with caution.  And if you come across any websites/articles/blogs that quantify PolitiFact's Truth-o-Meter rulings, please let me know in the comment section.

Announcing "Truth Index+"

While PolitiFact's Truth Index scoring methodology appropriately penalizes the lower scores, particularly Pants on Fire, it also distorts the overall score in a way that prevents valid comparisons from being made.   It basically takes comparisons out of proportion.   It is also more difficult to graphically present with its negatives.

 

Back in 2010, I began with a grading system called the Politi-Score, where a True was an A+ at 100, a Mostly True was a B at 90, down to a 50 for a flunking Pants on Fire.  That wasn't bad but it also made comparisons too "tight".  I then tried starting at 100 points working down to negative 25 for Pants on Fire, because others in a PolitiFact discussion group I presented my work in seemed to feel that the penalty was necessary as well.  About that time PolitiFact created its own measure called the Truth Index, which starts at 100 for True, makes Half True the zero and then goes down to negative 150 for Pants on Fire.  While I've stuck with that for a while now, I've realized in order to attempt things like standard deviations or z-scores, negatives might be difficult to work with.

 

This time I decided that True will be scored at 100 working down by equal 20 point increments to zero for Pants on Fire.  This will also make it easier for doing graphs by doing away with the negatives.  It also makes percentage comparisons easier to understand because the scores are perfectly proportional.  It is almost the equivalent of a percentage measure in and of itself.

 

So now a timeline can be developed from the beginning (2007) until the end of September of 2012.   I've also included the cumulative timeline for the Congressional Quarterly--the group that worked with PolitiFact when it first began in 2007, ending at the end of 2008.  This chart shows the Democrats Truth Index+ versus the Republicans, and the results aren't surprising considering everything else published on this blog.  The period measure is quarterly for the cumulative Truth Index until the end of 2008, to show the Congressional Quarterly Index, after which it is annual, so the chart would be less cluttered.

 

Click to enlarge: CQ rulings stop at end of 2008 when CQ-PolitiFact partnership ended

So today it can be said that based on 2,463 PolitiFact National rulings, it rates a Democrat about 29% more truthful on the Truth-o-Meter than a Republican.  And the largest drop in the Republicans Truth Index occurred from the end of 2008 until the end of 2009 (about 7%).  Whether it's because the Congressional Quarterly was no longer assisting is disputable--my feeling is it had something to do with the new Democrat president's term and the advent of healthcare reform in congress.

 

Next up will be a look at the "Pants on Fire" factor for the same data.

PolitiFact Rulings thru September, 2012

Reviewing the PolitiFact rulings almost one by one from since "in the beginning" to re-verify my totals and make the details more accurate has been quite the experience.  It's taken almost two months, and I found I made a lot of mistakes:  not enough to change all the averages I've presented in this blog, except in one area, which is the "Official/Non-Official."  I believe the well known  Eric Ostermeier study of 511 PolitiFact rulings from 2010 used those currently or previously in an elected office: I decided that was not a good way to do it because there may be a difference in the level of truthfulness between being in office and not being in office.  Office-holders were only designated as such when they were in an elected office at the time the statement was made as evaluated by PolitiFact.

 

I had a couple gender and party misidentifications,  plus quite a few records which were done by the state PolitiFacts which I had included with National for through 2010.  My ruling count now corresponds with what someone at PolitiFact kindly told me their number was.  So, through the end of September, 2012,  PolitiFact added a record number 556 rulings for the quarter, to now total 6,274 rulings.  Here is how they breakdown affiliation-wise:

 

 

There are a lot of reasons why PolitiFact has selected proportionately more Republican statements than Democrats, the main one being that outside the president, Republicans have dominated state/local governments, particularly the PolitiFact regions that started up in 2010, such as Ohio (John Kasich), Florida (Rick Scott), Virginia (Bob McDonnell), Texas (Rick Perry), and Wisconsin (Scott Walker).  If PolitiFact had a New York or California affiliate, I'm guessing we'd see more Democrat rulings.

 

Next the revised Truth Index--which I'm calling Truth Index+ for now-- will be explained featuring the new database.

Liberal Proof of PolitiFact (Liberal) Bias: A Summary Thus Far

A better standard is needed for me to say something at PolitiFact (PF) looks like "liberal" bias than for me to be a Republican and just not like what they say like many of its detractors.   Or to pick something out of selected rulings that I think is "bad journalism" when it's just my opinion.  Let's review what I've come across so far that I have assigned to "Liberal Proof of PolitiFact Liberal Bias" because I honestly have to admit there isn't much way around explaining it otherwise:

(1)   Yes, there is some inconsistency on the application of tax rates in the rulings, especially what exactly an "effective tax rate" should consist of, as noted in this recent post.   A ruling from PF Ohio and one from PF Rhode Island are particularly suspect for not being as thorough as they could have been and as a result appearing to  jack up the Truth-o-Meter for Democrats.

(2)  The PolitiFact National writers do seem to assign a lot more Pants on Fire to Republicans than their stateside counterparts (as well as produce rulings which make the Republicans look less truthful overall), with two exceptions.

(3) The Republican Party of Virginia was correct in one part of their over-the-top study of PolitiFact Virginia this past July:  it did appear to "dump" positive Republican rulings in the weekend Richmond Times Dispatch; however, they may be wrong assuming most people only read the news on weekdays.

(4) For the first three months of 2012, a weekly trend analysis of the Democrat and Republican Truth Index  appeared to indicate that at the end of the month, ratings would "even up."  I haven't done any analysis since, but have it set up in my data to do it for the year 2012.  What this means--those with a conspiracy mindset might think this was a deliberate attempt by PolitiFact to periodically even up the Truth-o-Meter (or the Truth Index) between parties.  I'm not sure what it means, and it might not occur again.

(5) Pat Boone's advertisement about Obamacare's IPAB rationing care got doubled up in Pants on Fire from two relatively-leaning-left-based-on-Truth-Index writers, but when a Georgia congress person made a very similar statement calling the rationing "the power to deny care to seniors" it received a Half True from PF Georgia.

(6)  In my version of what was to become my conservative counterpart's Pants on Fire factor, a measure of how much "more" Republicans are assigned Pants on Fire rulings, I looked at the Pants on Fire for PolitiFact's first 5,000 rulings and then proportioned them to the percentage of rulings overall.  The Republicans had 2.5 times as many Pants on Fire as Democrats, not considering the percentages.

7)  I also looked at the Truth Index in terms of writers,  by occupation (officials versus PACs, etc.) and by what I judged to be "local" rulings, for the year 2011, with varying results, although for the most part, they would predominantly favor the Democrats.  The one exception I found for what I judged to be "local" rulings, done for the first time in the final quarter of 2011,  was totally reversed in the first quarter of 2012.

8)  PolitiFact's most egregious problem may be "cookie cutter" rulings, where rulings are duplicated and serve to amplify the Truth Index negatively,  although the claimant often goes without political affiliation even though it's obvious what that affiliation probably is.    A major one this year was the across-the-board claim by "bloggers" that all the Democrats in congress were card-carrying socialists,  where almost all the PF states with a Democrat Congress member accused of "carrying the card" decided to jump on board and basically copy/paste the same ruling, inflicting a lot of Pants on Fire on those bloggers who had to be Republicans.  It could also be a subject as seemingly innocuous as light bulbs, a popular one in mid-2011.    Now it's been turned on its head a bit with the doubled up-assignment of "True" to a Democrat. 

Now, there's a few rulings of the same genre where PolitiFact has not been kind to the Democrats, and an excellent case of what could be termed Republican bias by PolitiFact Wisconsin.  And it would not take a lot of analyses to refute a few of my findings.  It just needs to be neatly summarized and put out there.

 

 

PolitiFact Detractors: Virginia Night Cap

It seems my conservative counterpart was forced to reference my analysis of the recent mega-paper protest by the Virginia Republican Party to PolitiFact (PF) Virginia, basically presenting an overly voluminous case of why it was biased toward Democrats. Bryan White called me a person with “semi-daily clock-work accuracy” as far as my being “correct” in terms of the ruling “statistics” used in the paper. Gee, I was hoping for a battle of wits, but I guess he’s apparently unarmed.

But White did bring out one point with regard to “the Weekend Dump.” This was the Republicans’ contention that more positive Republican PolitiFact rulings were “buried” in the weekend editions of the Richmond Times-Dispatch, while negative ones were saved for during the week, on the premise there were more readers during the week. He felt that the Sunday or weekend news may have more readers, not less, which contradicts their “bury” allegations.

Dumping stories over the weekend can put them in the Sunday newspaper, which is often the most widely read portion of a major newspaper. No case is made for the significance of a weekend dump for either a daily paper or an Internet news site. If the Richmond Times-Dispatch literally publishes the most positive Republican stories in its least popular editions then the Virginia GOP may have a legitimate gripe, but that evidence does not appear in this document.

Where readership demographics are reported on the internet, four major markets (New York, Chicago, Los Angeles and San Francisco) separate weekday and Sunday readership, and it appears Sunday is always higher for all adults. This backs up White’s assertion.

On the other hand, let’s consider the growing group of people who get their news on line. For example, (it appears to me that) many more comments are made on line at the PolitiFact Facebook page during the week, as the rulings are generated. Less commenting is posted on weekends, although some rulings are published late Friday. This may not mean anything, but this would seem to infer that the rulings are read by the majority of on line news fans during the week. Of course, the only way to verify that is for PolitiFact to look at its website “statcounter.”

Click to enlarge:  You can
barely make out Virginia since both
parties are so close to zero.
Compare to Wisc. and Nat'l.

In reviewing the Republican Party numerical data, I computed the Truth Index for ALL rulings ever done by PolitiFact Virginia to evaluate against the Republican’s anecdotal contention that the rulings “on average” favored the Democrats. Warren Fiske reported a very similar result to mine using a different scoring method which gave equal weight to the Truth-o-Meter categories. I used the scoring method delineated by Bill Adair for the Truth Index about a year ago, which more heavily weighted the negative rulings. But the result was approximately the same: the Republicans and Democrats were in a dead heat at PF Virginia. As an update, however, I started running overall stats for 2011 to current, as I have more detail for those records to make more interesting analyses. In charting PolitiFact by state, it was found that for 166 rulings done by PF Virginia since the start of 2011, there was still a dead heat, but the Republicans were trending to a very slight advantage over the Democrats. In other words, the very slight advantage the Democrats have in their overall score was accomplished in 2010, when PolitiFact Virginia first kicked off, and then it reversed to the Republicans' favor since that time. But it’s more of a “margin of error” variance, so it does not mean much.

The snippet to the left shows the Democrats in blue and the Republicans in red.  I will be publishing the complete chart in a later post.  The baseline is "zero" on the PolitiFact Truth-o-Meter, a rating of Half True.  Each line is 10 points on the Truth Index.  The Truth Index score was rounded for easier reading; in Virginia the Democrats were at a positive .78 while the Republicans were at a positive 1.56.  It should be noted that PolitiFacts Virginia and Wisconsin are the only two PolitiFact states to appear to favor the Republicans; all others have higher scores for the Democrats, as can be seen in the overall score at the right, where for 2,935 rulings, the Democrats are at a negative .81 and the Republicans are at negative 24.07.  (Perhaps the Virginia Republicans are looking in the wrong place for "bias" and this is something the Republican party should do on a national level.)

Nevertheless, White makes a brief admittance at the end of his post that the Virginia Republican Party’s paper was similar to his PolitiFact Bias blog: “… the report scores with the anecdotes and not much else.” He notes the last 32 pages of the 86-page opus were fluff, a simple listing of all the Republican rulings by PolitiFact Virginia. In other words, it’s right there in the same league as Eric Levine so aptly described: the Republican Party of Virginia report on PolitiFact is just one more Gish Gallop of anti-PolitiFact propaganda.

Postscript 7/24/2012:  I should also note that this puts a previous post on the weekday/weekend publishing of PolitiFact National in a different light.  It makes them look good from the aspect of the newpaper, but not so good from the aspect of on-line readers, in terms of a alleged "bias."

Truth Index: Don't Rush Me

After publishing the first half PolitiFact (PF) ruling results, it’s come time for more breakdown of the detail, and as I accomplished that I came across an “amusing” error I had made: I accidentally classified Rush Limbaugh as a Democrat! Oh woes me! So that’s been fixed. He only had one Pants on Fire ruling for the first half of 2012, but it did slightly affect the Truth Index.

Here’s how it breaks down by “source”—starting with the President/VP, the congress, the governors, the states and local governments, as well as advocacy groups, PACs and the media, and includes the overall Truth Index (here's the one for first quarter of 2012). “Others” includes former officials, blogs, chain e-mails and websites. A new category has been added called “Candidates (other than President)” as we have more and more of them due to the beginning of the 2012 election campaign. All officials I’ve categorized are “sitting”: in the case of presidential (I’d say ex-presidential but I don’t want to incur the wrath of Ron Paul fans) candidates like Ron Paul and Rick Perry, they are included in the categories of their “sitting” positions, congressman and governor.

Click to enlarge:  Groups (PACs etc) have a far lower Truth Index (-43) than individuals (-12) .

The following is a further breakdown of the pie chart, showing Democrats and Republicans. As can be seen the “local” category is now almost exactly even between Democrats and Republicans, whereas for the first quarter I showed a Republican advantage. The difference in the “local” Truth Index you’ve seen in earlier posts (as far as number of rulings) is that the chart indicates local politicians only, but there are also local issues covered by almost every source category which was what my earlier posts were about.

Click to enlarge:  FIRST HALF 2012: Only 3 rulings on Democrat governors made for the lowest Truth Index.

The senators and governors having more positive rulings than the other positions seems to be holding. The senators’ longer terms and their proverbial meaning of being “the eldest and wisest” (although let’s remember the word senator is from the same root as “senile” ha ha!) may have something to do with their being more “truthier” when subjected to the Truth-o-Meter. I posted something earlier on the predominantly Republican governors as far as who was the lowest on the Truth Index (Scott Walker of Wisconsin) but relatively speaking as a group they are much more “truthier” than their political counterparts.  

Next up, PolitiFact states or maybe PolitiFact writers?

Truth Index: Local Update

In the last quarter of 2011, in my quest to find unique statistical trends from the data mining I had done of PolitiFact’s Truth-o-Meter rulings, I had assigned a field called “Local” to each of the PolitiFact rulings. As I described at the time:

What is termed “local” deals with the nature of the PolitiFact (PF) ruling. Most often it would show in the title or subtitle. It means it just relates to something having to do with the state or locality, that it’s not national. It could be ideological, but it still pertains to a local issue. None of these rulings were from PolitiFact National. Intuitively they wouldn’t, but state PolitiFact rulings can be done for an office holder of that (PolitiFact) state, while the subject of the fact-check is of national scope. In other words, without reviewing each ruling, the state rulings can't all be counted as local…

…Republicans are more truthful than Democrats….by a pretty good margin. Republicans were at a precise “zero” on the Truth Index, while the Democrats were a negative 27. There were 173 “local” rulings which represented 38% of all the PF rulings in the fourth quarter of 2011; Democrats and Republicans were also equally represented, each having 74 rulings. There were 25 rulings for which the affiliation was independent, non-partisan or just unknown.

That quarter I had 174 rulings of a local nature, and I got a somewhat surprising result in the Truth Index by party: the Republicans, as noted in the quote above from my post, were ahead, way ahead of the Democrats on the Truth Index (as compared to the Democrats normally being higher, as I have shown when you look at all rulings accumulated). I wondered whether it just an anomaly for the quarter. Although I was excited at this seeming incongruence, and confounded to determine an explanation, I still needed more data in order to confirm that this trend continued.

Click to enlarge:  The Republicans local "Pants on Fire" quintupled from 2011 to 2012, lowering their overall Truth Index.

So I now have six months of data to look at, almost 400 rulings (more than twice what I had the first time) and it seems my initial findings may have been premature….or maybe the Truth Index is functioning the way it should. In the first quarter of 2012, the Democrats have reversed the trend from the last quarter of 2011, and the Republicans reversed as well. The Democrats went from a negative 27 in the 2011 fourth quarter to a positive 11 in the 2012 first quarter for an overall average of negative 5.6. The Republicans went from an exact zero to a negative 12 for an overall average of negative 6.8. In other words, the Democrats and Republicans are statistically tied for the Truth Index average at around a negative 7 based on six months of Truth-o-Meter rulings. Compare this to a negative 13 for all rulings (well, the first 5,000) or a negative 9 for individuals.

But this also serves to prove in a sense that PolitiFact isn’t doing anything deliberately—the Truth-o-Meter goes up and it goes down, and it has nothing to do with “selecting” those statements which would benefit either side.

One of the reasons the Republicans “took a fall” this quarter was that there was a steep increase for them in Pants on Fire rulings, from two in the last quarter of 2011, to ten thus far in 2012. The pantalones en fuego ranged from such things as a statement by New Jersey Governor Chris Christie that “23 percent of kids starting ninth grade in Newark will graduate in four years” to PolitiFact Tennesee’s ruling on state representative Frank Niceley on his claim regarding the Tennessee Wildlife Resources Agency bringing coyotes into the state "to try to keep the deer population down." They were spread out over almost every PolitiFact state and didn’t all occur at any particular time during the quarter.

But this does show, so far, that overall, politicians appear to be a bit more truthful at a local level. I will continue to keep the “local” factor in the rulings and monitor them to get a better picture.

Truth Index: 2012 First Quarter

For the first three months of 2012, PolitiFact labored to produce 516 Truth-o-Meter rulings.  This brings the grand total number of rulings, since PolitiFact began, up to 5,223 as of March 31 (with the milestone 5,000 passed in the latter half of February).  Since I started tracking the Truth-o-Meter ruling by ruling beginning in 2011, this is the highest number of PolitiFact rulings done in one quarter.  The last time 500 was exceeded was in the second quarter of 2011.   The addition of PolitiFact Tennessee and the work of PolitiFact New Hampshire supplementing PolitiFact National helped put them over 500.

 The Truth Index for all the rulings has changed very little from the overall Truth Index rating at the end of 2011, of negative 13.3; it's gone up microscopically to a negative 13.2, and indicates a very flat trend line up (sort of like the downward trend of national unemployment, ha ha).  Here is how the distribution of the rulings by party broke down: 

This breakdown has improved somewhat since the end of 2011 when fact-checks on Republicans reached a heady 67 percent while only 24 percent were on Democrats and Liberals in December.  Compared to overall, however, there has still been more emphasis on Republican fact-checks (overall for Republicans it's about 53 percent and for Democrats, 39 percent).  The obvious explanation, of course, is the coverage of the Republican primary candidates as well as the domination of Republican governors/governments in the states.

Click to enlarge.

The Truth Index "Report Cards" for the quarter (above) show the Republicans held steady, moving up a point at a negative 22 while the Democrats improved very slightly from about a positive 1 to just over 5.  As a point of reference, the Republicans' score shows a very average spread of Truth-o-Meter rulings (one in each category) while the Democrats, at a score close to zero, are slightly better than average, as if they were spread equally across all categories except Pants on Fire.

Next up is a new and improved feature:  instead of a month to month Truth Index, it will be week to week for the first quarter...although it doesn't tell you much.

Truth Index: Point of Reference

Since starting the tracking of the PolitiFact Truth Index, I’ve often wrestled with baseline and relative reference numbers in trying to give it a relative measure. I often use the average PolitiFact Truth Index overall (for PolitiFact’s first 5,000 rulings, it was -13.4). In the charts I create I often point out the Half True indicator, which is zero. And of course, one can compare the truth average of one group against the other.

For 2012, I created something I called an “Insta-Chart” so for certain things I’d instantly have a PolitiFact Truth Index App type chart to which all I had to add was a title. This would also make most of my charts appear consistent. And of course, when you’re doing it by formula you want to test your formula to make sure it works. So I put a 1 in for all the Truth-o-Meter categories, and in doing so the thought occurred to me that this “across-the-board” Truth Index might be THE baseline.

If someone PolitiFact rated all the time had, for whatever reason, an equal number of statements in each of the Truth-o-Meter categories, their Truth Index would be a -25. This reflects the fact that the Truth Index is a weighted average. In that respect, I have made a big mistake in looking at the individual Truth Index in terms of the Truth-o-Meter categories, for example, when I say the Republicans, with a Truth Index of -23, were on average about half way between Half True and Mostly False.

Here is how the table would look (and how the weighting affects the averages) in terms of having 1 in each of the six Truth-o-Meter categories as the baseline “average”.

Truth Index	Means in "Truthfulness"	Equates to in Score
50	George Washington Wannabe	1 in every category except Half True, Mostly False or Pants on Fire)
25	Way above average	1 in every category except Mostly False or Pants on Fire)        .
0 (zero)	Above average	1 in every category except Pants on Fire
-25	AVERAGE	1 in every category
-50	Below average	1 in every category except True.
-75	Way below average	1 in every category except True and Mostly True

Instead the Republican average of -23 should be compared to the generic across the board Truth Index average of -25. This means the Republicans are about evenly divided among all the Truth-o-Meter categories. The Democrats, at a Truth Index average of just above zero, are evenly divided among the True through False categories "without" Pants on Fire.  So we have a measure which incorporates the weighting of the average.

I don’t know if this is how it should be presented, but it does provide a comparison, and I will be using this as a reference point in future postings.

Truth Index: 5,000 Rulings Continued

The various permutations I show on the 5,000 PolitiFact (PF) rulings are very similar to my series on the 4,707 rulings analyzed through the end of 2011, so I will not repeat them here. The 293 rulings which took it up to 5,000 represent about six percent of all the rulings, so the stats should be very close to what they were then. I do want to show, however, that there are some trends which seem to continue that I didn’t emphasize in my end of 2011 analysis.

Affiliation.  The majority of rulings  (53%) were on Republicans/Conservatives/Libertarians; 39% were Democrats/Liberals, and the rest were non-partisan or unknown (see graphic above).  PolitiFact's own numbers, if they have them, would differ from mine slightly because of more delineation of the affiliation on my part:  for example, their chain e-mails are not broken down by party (although many of them can't be) and there are a few people for whom I determined affiliations for which PolitiFact had assigned a "none."

PF's table adds to 99%

The National percentages. I confirmed my agreement with the National Truth-o-Meter rulings which were displayed in the 5,000 ruling celebration article—that is, that it was very close to what I showed for 5,000 rulings including the states. However, it’s important to match to the National “only” rulings to see how I compared apples to apples, and there I have a slight discrepancy. For National, all percentages match except I am one point higher on the Pants on Fire, at 9 percent instead of 8 percent. Now, I may have more rulings in the National number because of the way I added up the rulings through 2010, which may have caused the over-calculation, but it is also important to note that the numbers as given by PolitiFact add to .99, not to 1.00. Carrying it out to decimals, I have 8.69% for Pants on Fire for PolitiFact National. For greater analysis on PolitiFact regions (by party) through the end of 2011, click here.

Individuals versus Groups and “Entities.” Some of these are fairly intuitive when you think about it. Individuals (85% of the 5,000 rulings) score more truthful on the PolitiFact Truth Index than groups or “entities”(advocacy, party boosters, bloggers, chain e-mails, etc.) They average a -9.06 while groups and entities are -37.55, a pretty substantial difference. For a further breakdown of these groups and how they compared affiliation-wise, click here.

Individuals comprise about 85% of PolitiFact's 5,000 rulings and are more "truthful" as measured by its Truth Index.

Office-holders (current and former) are 74% of the 5,000 with a Truth Index of -8.47 while non-office holders (which would include those groups and entities mentioned above) score a -27.77 on the Truth Index.

And as noted in a previous post, one out of seven (674) of the 5,000 rulings were on females who had a lower aggregate Truth Index (-13.2) than the males (-8.28) although it is a fairly small variation.

All suggestions for further refinements are welcome!

Truth Index: Smart Politics Revisited

Early last year I did a pretty detailed analysis on Eric Ostermeier’s study of about a years’ worth of PolitiFact (PF) National rulings. I compared my larger numbers to his at that time; to be specific, Ostermeier evaluated 511 PolitiFact statements while my sample was 1,784. My number actually included everything that was in his 511.

This time I am again looking at the entire population of PolitiFact rulings to compare to Eric Ostermeier’s. The most notable and most cited by various conservative pundits in terms of its possible bias is the 76/22 one: that is, 76% of all False and Pants on Fire rulings are attributable to Republicans while only 22% are attributable to Democrats. Mark Hemingway of the conservative Weekly Standard cited it in a recent NPR radio piece with Washington Post Fact-Checker author Glenn Kessler.  Here's Eric's finding:

Republican statements were graded in the dreaded "false" and "pants on fire" categories 39 percent of the time, compared to just 12 percent for statements made by Democrats.

That means a supermajority of falsehoods documented by PolitiFact over the last year - 76 percent - were attributed to Republicans, with just 22 percent of such statements coming from Democrats.

So, how does it look now? I could never get close to Ostermeier’s numbers, although Republican False and Pants on Fire always outnumbered the Democrats. I did it in several ways to compare to Ostermeier: one with everything, which would include the PF states, one with everything including the non-office holders, and one just on National. I actually achieved the 22% for the Democrats by using the top 60 in terms of those with 10 or more rulings as posted previously on individual Truth Index average; but that is not a representative sample. The Republicans in this case were just over 70%, not 76%. But in all the other more inclusive measures, as shown in the graph below, the Republicans varied from around 60% (all False and Pants on Fire rulings) to 69% (PF National only). The Democrats were anywhere from 26% to almost 30% for all False and Pants on Fire rulings. Here’s a graphic of all my “permutations,” including one which included everyone and everything: for the 4,707 rulings PF did from the day it was “born” through the end of 2011, the False and Pants on Fire categories were roughly 60% attributable to Republicans, 30% to Democrats, with the remaining 10% for Independents, non-partisans and unknowns. While it’s still a 2 to 1 ratio in favor of the Democrats, it’s clearly better than the 3.5 to 1 as found by Ostermeier.

Click to enlarge:  Smart Politics' percentages for Republican "Falsehoods" exceeded all five ways I did it with the entire PF database. (NH is New Hampshire, which appears to only do national rulings related to the 2012 Elections.)

The other Truth-o-Meter categories were reviewed by Ostermeier as well, and so I obliged with my own, calculating not only just the percent of total rulings compared, but the rulings as a percent of all those in the Truth-o-meter rating. Again, this was for elected officials and those holding office. According to Ostermeier, "....… Democrats have therefore been presented as much more truthful - with over 75 percent of statements receiving the top three grades of True (16 percent), Mostly True (27 percent), or Half True (33 percent)."

The numbers indicate that Ostermeier’s sample appeared to be skewed more in favor of Democrats, no matter how I tried to shake the numbers. I had 10 fewer points for the Democrats and 5 more for the Republicans, and there was even one stat in favor of the Republicans: that was the percent True, Mostly True and Half True as a percent of the total True, Mostly True, and Half True: Of the 2,033 rulings in those Truth-o-Meter categories, 50% were attributable to Republicans, while 47% were attributable to Democrats. Part of the reason may be, of course, that there were more Republican rulings. While at the time Ostermeier found that PF National had roughly the same number of rulings for each party in his sample, they began to become more heavily Republican with the 2010 Elections putting more Republicans in office, and the 2011 Republican presidential campaigns and debates requiring frequent fact-checks. As a result, Republican rulings increased about 10 percent while Democrat rulings dropped almost 14 percent from Ostermeier’s more even percentages.

Next, and last but not least, a few fact-ettes and fact-oids shared from my database of PolitiFact rulings.