One Way PolitiFact Contends with "Selection Bias"

Coming from that exciting (ha ha ha) world of accounting, the audit or financial internal control function served to police those who kept the books, from committing fraud or gross accounting malpractice.  There were certain procedures that had to put into place:  one was not to let one person control too much of the process, and to transfer employees between jobs frequently--even to compel them to go on vacation because often when the employee was away, such fraud or malpractice would be discovered.

 

This analogy is of course not about "fraud" in relation to PolitiFact, but about how a journalist who in the same way might write all fact-checks for one person (on one topic) might encourage the potential for the selection bias that is so often cited by its many critics.  And in the course of my recording over 6,500 PolitiFact rulings I noticed a pattern in assignment of the fact-checks in that no one writer dominated covering any one person or subject.  I have previously posted that writer Lou Jacobson appeared to be the "economics" guy, but economics has a wide umbrella covering many subject areas, such as jobs, taxes and the federal budget, even healthcare.  For example, a selection of the 13 consecutive rulings on Obama under the subject of taxes, out of 121 total rulings on Obama in 2012, shows this alternating pattern of ruling assignment quite clearly:

Click to enlarge:  Note column for writers. 

Even in the PolitiFact states, alternating topics and who was assigned the fact-check seemed to be the norm.  Here's an example from the  fact-checks by PolitiFact Wisconsin of its most frequently checked person Governor Scott Walker, on one of its most frequently checked topics, the state budget:

Click to enlarge

On occasion where there was a speech or television interview with a lot of statements to check, we might see a concentration of fact-checks by one writer, sometimes as many as five, but this was rare.  Eventually another writer would take the reins.  After the election in some of the states, it appeared that a single writer took over until the new year; then it was back to alternating.

Of course, I can't say that this resolves the problem of selection bias.  The type of bias that they're being accused of having--which involves a liberal partisanship--would still be evident regardless of how and to whom a fact-check is assigned, if it could be proved.  But I would think it would be somewhat ameloriated through multiple journalists covering multiple topics, because not everyone has the same bias going in the same direction.

A Case for PF Wisconsin Right Wing Bias

While some try as best they can to prove that PolitiFact National has a left wing bias, there are a number of good reasons why it most likely doesn’t exist at national level.

1. The selection bias allegation is assumed to be only a party-related bias, contrary to PolitiFact’s repeated claim that is a newsworthy bias, and that ALL PolitiFact National writers/researchers/editors are liberal (and have dispensed with journalistic training in objectivity), for which one of its most vocal critics has provided NO evidence except for his own opinions of an anecdotal review of its rulings,  tainted with his own admitted conservative bias.

2. That other fact checkers, such as FactCheck.Org and the Washington Post FactChecker, check many of the same claims and (about 99% of the time) make similar findings, so therefore, it would have to be proven they are liberally biased as well in reaching the same conclusions.

3. There appears to be no obvious agenda cited by the critics except the sweeping generalization of selecting only statements favorable to Democrats and not favorable to Republicans.

IN WISCONSIN, however, using the above three points, there’s a very compelling situation which makes a good case for Republican (or right-wing) selection bias: .

1. A limited number of writers (they could be biased—but again, there’s no evidence they are, but it seems it would be less possible where there are more than just a few writers, as at PolitiFact (PF) National).  As to the question "who checks the fact-checkers" PolitiFact Wisconsin Senior Editor Greg Borowski could only say "all of our sources are listed and linked.  So check away."  But is having sources listed sufficient?

2. No one else fact-checking same claims because they are local to Wisconsin. .

3. An obvious agenda (the recall election) reflected by a Pants on Fire on the Democrat contender for governor on the eve of the election, with no claims checked on the Republican, and a few days before, particularly June 1, claims reported rated much more positively for Republicans.

4. And I’ve already noted that Republican-leaning news organizations in the PolitiFact domain tend to have truth indexes which favor the Republicans more. The Milwaukee Journal Sentinel, which sponsors PolitiFact Wisconsin, is a somewhat mixed publication in its endorsements: it endorsed Walker, but also endorsed Obama in 2008 and did not endorse either Bush or Kerry in 2004.

5. Individual Truth Index trends: Prior to the 3 month period before the June 5 Recall, Scott Walker’s Truth Index was a negative 43, while in that three month period it improved modestly to a negative 21. Tom Barrett’s Truth Index up until that three month period was a negative 9. In the three months prior to the Recall, it plummeted to a negative 62. Mind you, this was for only eight fact-checks.

Politi-Score: Bottom line of Fudging the Truth

The methodology for doing the above stats: Went through PolitiFact (PF) ruling page by Person/Group and included those with 5 or more PF Truthometer rulings . True ratings= 100 points, Mostly True= 90, Half True= 80, Barely True= 70, False= 60, and Pants on Fire= 50. The number of points was then multipled by the number of rulings made for the category, the points added up, and then divided by the total number of rulings.

"People" or individuals (not groups/committees/associations or emails) were only included here , and nothing from PF's "Flip-o-meter."  They are all current or former office-holders/strategists/consultants.  The pundits (those who have radio/television shows and are considered more or less "entertainers") were separately scored.