Grading PolitiFact *Liberal*-Style: No "Set" for Navratilova

But another "point" perhaps!

Martina Navratilova's recent statement (published May 7) rated by PolitiFact as a Half True resulted in a big brouhaha review from MSNBC'S popular pundit and gay-rights promoter, Rachel Maddow. Navratilova stated on the TV show "Face the Nation" that "in 29 states in this country, you can still get fired for not just being gay but if your employer thinks that you`re gay, you could still get fired.”

On the face of it, it is true that 29 states do not ban discrimination against gays. But PolitiFact's ruling took into account that in those 29 states there existed similar protections for some employees by means of federal and local laws as well as private company policies.

I had some problems, however, with how PolitiFact came to concluding this Half True...I would want to ask writer Louis Jacobson a couple questions: Did he ask Navratilova or any representative of Navratilova how she framed it? Could it be that she was not aware of such exceptions? It was not indicated in the sources. Did he "count" in terms of those who were in the "exception" category, approximating what percentage of people would be covered in those states not having such protections? It could have been estimated, but it would have taken some time. For example, there's Virginia, which has a high percentage of people working for the government but as a state does not prohibit such discrimination. But in other states, even with other protections, the percent of those having them might be very small--so small that it would not be that relevant.

 

I was a bit disappointed in how Rachel Maddow did the segment on how PolitiFact handled the ruling. Her very curtailed explanation of the exceptions was as follows:

They fact checked a statement about state law, found it to be true, decided it didn’t seem seemly or whatever to actually just call it true, then they searched other unrelated information about how there are other kind of things, besides states, like some companies, they don’t want to discriminate, and doesn’t that count for something?

I vehemently disagree with Maddow. It IS related, and they're actually the same types of laws (which she conveniently left out), not just "some companies" that "don't want to discriminate." It was a four- point bullet list (which some like to call "concrete examples").

Erik Wemple of the Washington Post describes Maddow's approach to fact-checking as "strict constructionist" which means "don't look at the context." Well, to take a term from tennis (used in another way), I'd love, LOVE to see what Maddow would do if PolitiFact did the same with, say, the 2012 PolitiFact Lie of the Year from Mitt Romney about Chrysler moving  vehicle production to China. Perhaps then she'd understand why it is important.

 

Wemple noted that it was not Navratilova's intent to mislead. But the Half True is not a ruling given for misleading statements; it is for those "missing important information." This is where I diverge further however, because the statement on its face was fully accurate, not partially, the criteria for a Half True. As it was fully accurate, but needed the additional information for the other ways in which discrimination was banned, it really qualifies for the Mostly True.

 

Because it would be difficult to account for the number of those who would benefit from those other laws banning discrimination, and because context was not verified by speaking with her, it would have been entirely acceptable to give Navratilova a Mostly True instead of Half True. The exceptions put a large shadow on a True ruling, and PolitiFact was correct in not awarding it. Half True, however, may have been too severe.

One Way PolitiFact Contends with "Selection Bias"

Coming from that exciting (ha ha ha) world of accounting, the audit or financial internal control function served to police those who kept the books, from committing fraud or gross accounting malpractice.  There were certain procedures that had to put into place:  one was not to let one person control too much of the process, and to transfer employees between jobs frequently--even to compel them to go on vacation because often when the employee was away, such fraud or malpractice would be discovered.

 

This analogy is of course not about "fraud" in relation to PolitiFact, but about how a journalist who in the same way might write all fact-checks for one person (on one topic) might encourage the potential for the selection bias that is so often cited by its many critics.  And in the course of my recording over 6,500 PolitiFact rulings I noticed a pattern in assignment of the fact-checks in that no one writer dominated covering any one person or subject.  I have previously posted that writer Lou Jacobson appeared to be the "economics" guy, but economics has a wide umbrella covering many subject areas, such as jobs, taxes and the federal budget, even healthcare.  For example, a selection of the 13 consecutive rulings on Obama under the subject of taxes, out of 121 total rulings on Obama in 2012, shows this alternating pattern of ruling assignment quite clearly:

Click to enlarge:  Note column for writers. 

Even in the PolitiFact states, alternating topics and who was assigned the fact-check seemed to be the norm.  Here's an example from the  fact-checks by PolitiFact Wisconsin of its most frequently checked person Governor Scott Walker, on one of its most frequently checked topics, the state budget:

Click to enlarge

On occasion where there was a speech or television interview with a lot of statements to check, we might see a concentration of fact-checks by one writer, sometimes as many as five, but this was rare.  Eventually another writer would take the reins.  After the election in some of the states, it appeared that a single writer took over until the new year; then it was back to alternating.

Of course, I can't say that this resolves the problem of selection bias.  The type of bias that they're being accused of having--which involves a liberal partisanship--would still be evident regardless of how and to whom a fact-check is assigned, if it could be proved.  But I would think it would be somewhat ameloriated through multiple journalists covering multiple topics, because not everyone has the same bias going in the same direction.

Truth Index+: The Top Ten PolitiFact Writers

There are ten PolitiFact (PF) writers who have produced just about half of PolitiFact's rulings, from its beginnings in 2007 up until I ended my compilation on Election Eve, 2012.  Another roughly 160 writers composed the rest of them, many serving multiple functions of writer, researcher and editor.  There were 47 writers who wrote just one to two rulings each.

 

The highest producer by far is Lou Jacobson, whose 676 fact-checks (some done in association with other writers at PolitiFact) make up a bit over 10% of PolitiFact's total output.  Number 2 is Angie Drobnic Holan, now the Editor at PolitiFact Florida, at 571.

 

Below is the Truth Index+  for the Top Ten, with scores for Republican versus Democrat.  The best way to look at these averages in terms of "bias" or "lean" would be the percent variance in the Truth Index+ between party for each writer.  Lou Jacobson, Angie Drobnic Holan, Robert Farley and PF Georgia's Willoughby Mariano have about a 22% variance leaning toward the Democrats, only to be exceeded by Number One C. Eugene Emery of PolitiFact Rhode Island, who has a whopping 38% Democrat lean.   Tom Kertscher at PolitiFact Wisconsin swings 22% in the opposite direction; the remainder go from 3% in favor of the Republicans to around 10% in favor of the Democrats.

 

Click to enlarge:  NA in parentheses = PF National; then number of rulings written.

So, what can we learn from the Truth Index+ for each writer?  I decided to take a closer look at the top four writers and an "outlier" in terms of some of the items I had keyed in:  party, titles, people, subjects.

 

Lou Jacobson, for example, was No. 5 in "lean" variance as noted above, although other than Emery he was in a virtual tie with four others.  But he had a few features in his 676 rulings that stood out:  he is definitely "the economics guy" on the PolitiFact team.   The subject of about 40% of his rulings was a fiscal one: economy, federal budget, stimulus, taxes etc..  This was a far larger percentage than any other writer.  Healthcare was the subject of 25% of his rulings.    In terms of "occupation" his being on PolitiFact National's team meant that about 30% of his rulings were on the president and vice president and the presidential/vice presidential candidates (in 2008 and 2012).

 

A check on names revealed a notable person who seems one of Jacobson's "fact-check favorites"--Michele Bachmann.  He did almost a third (17) of her 54 rulings, and gave her 5 of her 12 Pants on Fire awards, as well as five Falses (in other words, when a claim was found False for Bachmann, Jacobson determined it was Pants on Fire half of the time.)  In comparison, Angie Drobnic Holan only gave Bachmann one Pants on Fire out of five rulings found False.

Lil White Lies: I Dream of Gini

Lawyers’ Rules of Thumb(i.e., how my conservative counterpart works his blog):
If you can’t use the law (i.e., argue fallacies), use the facts
If you don’t have facts, use the law (i.e., argue fallacies)
If you have neither law (fallacies) nor the facts, pound on the table and create a distraction.

Here we have another sublimely bloviating multiple-infected specimen of misrepresenting the source along with the underlying argument of a PolitiFact (PF) ruling. My conservative counterpart takes aim at a statement made by ex-Florida-congressman Alan Grayson on MSNBC’s Rachel Maddow Show that there are “are only five countries in the entire planet that are more unequal than the United States in the distribution of our wealth.”

The first part of Bryan White’s argument is that PolitiFact’s Lou Jacobson (who else!) incorrectly attributed where Grayson said he got his information: a link provided in Wikipedia to a “peer-reviewed” paper published in The Economic Journal in 2010. He explains:

If the PolitiFact team had paid closer attention, they would have noticed that the numbers on which Grayson relies were not published in any peer-reviewed journal. The numbers Grayson used ultimately came from an appendix (Excel file) to a working paper that preceded the version published in The Economic Journal.

PolitiFact reports falsely in claiming that PolitiFact sourced its numbers to the linked "a peer-reviewed paper." Sift through the linked work from now until doomsday and you won't find "Namibia" mentioned at all nor any comprehensive list of Gini coefficients (identified by PolitiFact as the primary statistic used to identify inequality) for all the nations of the earth.

But if one reads the paper (paid closer attention!), one would find 25 countries listed,with the identical Gini Coefficients from the “unpublished” Appendix VI, in Table 7 on page 246 . The writers describe how they chose these 25 countries: “Note. Countries are listed according to the number of members of the global top wealth decile.” In other words, the writers made this the criteria for selecting countries from Appendix VI. In fact, Table 7 has the U.S. as Numero Uno in wealth disparity, as the four countries which exceed it are not there, although one of them, Switzerland, is mentioned on the previous page: “…Switzerland, whose distribution is based on hard data with good detail in the upper tail, has a Gini coefficient of 0.803 (Davies et al., 2009, Appendix VI) The US and Switzerland have Gini coefficients close to the global wealth Gini of 0.802.” And right there is the cross reference to “Davies et al., Appendix VI” which is cited frequently throughout the paper. James Davies is the first “academic” listed as the writer of the paper.

So, if Appendix VI wasn't published, what becomes of this? Does that make it incorrect?

If one looks at the working papers cited/linked in a bottom reference at Wikipedia, one can see it reads like the paper submitted to The Economic Journal. In other words, it’s virtually certain that Appendix VI served as the data source for Table 7. Actually, it WAS published, although not in The Economic Journal: it was at the NBER, National Bureau of Economic Research. I suppose only the reference was required for the table instead of appending it to what was published in The Economic Journal; but for all intents and purposes it was the foundation of the paper’s Table 7 and was the sole reliance for the Gini data presented . You can say it was partially published, but it can’t be said, as Bryan’s trying to claim, that it wasn’t published at all.

Politi-Score: First Half Preview--PolitiFact Writers Revisited

The first half of 2011 is almost over, so I have started compiling PolitiFact rulings from where I left off at the end of March. As I start with PolitiFact National and there’s just a few days left in June, I thought it would be worth it to take another look at the PolitiFact National writers again in terms of the Politi-Score. At that time, it appeared that PolitiFact writer Angie Drobnic Holan favored the Democrats more than the two other writers Lou Jacobson and Robert Farley; however, now that more rulings have been accumulated, that theory may have been a bit premature.

Holan’s Politi-Score dropped from 63 to 53 for Democrats and went up a bit for Republicans; both Farley and Jacobson dropped in “truthiness” on both sides. Here is the chart for the first quarter, and below that (outlined in red) is the chart for the first half (first and second quarters combined):

First quarter 2011 (click to enlarge).

First Half 2011 (click to enlarge).

Conclusions? Well, if we are to believe that PolitiFact is liberally biased, it would seem they selected more Republican statements in the second quarter that were false. They did select more Republican statements, aside from whether they “knew” the statements would be ruled False or Pants on Fire. For the record, there were 116 rulings total published by PolitiFact National for the second quarter of 2011 (as of Monday, June 27). Of these, 45 were Democratic/Liberal statements and 67 were Republican/Conservative (the rest were non-partisan). The Democrats had only two Pants on Fire rulings, and 10 False; the Republicans had 11 Pants on Fire rulings, and 19 False. That’s a ratio of 5:1 on Pants on Fire and 2:1 on False comparing Republicans to Democrats.

Yesterday, a Facebook commenter named Vic Pilkington addressed me on something similar:

… what is your tally when comparing right and left "pants-on-fire" and "false" starements? I contend the right have a 4:1 edge over the left on those statements considered as having no basis in reality.

Well, I guess it’s even worse than Vic thought. For the second quarter of 2011, at PolitiFact National, there IS some basis for that reality. On the other hand, we have about a quarter of all the rulings represented here, and the rest (the PolitiFact partners) need to be reviewed.

Politi-Score: Angie's List Left

Last January I did a post in which I looked at Bryan White's Grading PolitiFact series of criticisms which he believes establishes a "preponderance of anecdotal evidence" of PolitiFact's liberal bias.  The title of my particular post was "Does Lou Lou do More Doo Doo" because, as I noted in the piece, Bryan has never given a "grade" to PolitiFact's writer Louis Jacobson any higher than an F.  Indeed, it appears he looks at more PolitiFact rulings by Jacobson than anyone else, although my count at the time revealed that Louis Jacobson was tied overall with Robert Farley at 23 rulings each, while Angie Drobnic-Holan came in second with 18 rulings reviewed.

Starting this year in my "counts" I decided to include the writer in order to calculate a Politi-Score by PolitiFact writer.  The quarterly results offered a slight surprise:  it may be that Bryan is not looking at the right person.  Lou Lou doesn't do the do do as much as Angie does (if there's to be a suspicion of liberal bias), according to the Politi-Score:

Click to enlarge.

This is roughly 100 rulings: Farley and Holan did one quarter each, and Jacobson did the other half.  The Politi-Scores tell a story, however, with Angie Drobnic-Holan giving Democrats a very high (the highest of all three) Politi-Score of 63 compared to her two PolitiFact National co-workers.

At this point, all this tells me is I need to continue compiling results for the writers to get a larger sampling of rulings, which would give more validation to this result.  If I have the time I may go back and compile them for 2010.  Stay tuned.

Lil White Lies: Does Lou Lou do more Doo doo?

As part of an analysis of PolitiFact’s (PF) “bias” I made a list of possible items to consider: one was, is there one PolitiFact writer/researcher who may be more biased or prone to error favorable to one side than the others? I thought Bryan White’s Grading PolitiFact series might provide some insight into that. I also wondered, now that I had a population numeric for PolitiFact rulings (2,788), did Bryan attain a number of grades that might be considered a sufficient sample. There was also a certain curiosity as to whether any other characteristics (who made the statements, etc.) of his sample revealed any other trends of interest.

So I went back to his blog and started adding up the rulings. I found that he “officially” started his Grading PolitiFact series in 2009 coinciding with the beginning of Barack Obama’s presidential term. So I counted two years forward from that, to January 20, 2011, getting a total of 97 rulings. According to statistical sampling techniques, he would need a sample of at least 142 rulings with the same finding of “bias” to achieve a 95% confidence level for that finding to exist in the rest of the population. Bryan also “graded” PolitiFact in 2008, so there are additional rulings on top of his 97. But I would make an educated guess that he doesn’t have 45 additional rulings to get to the 142 benchmark. Additionally we get brought down by an important (and perhaps most crucial) requirement of the sampling: it must be unbiased, and we know it is already tainted (because Bryan openly admits to his conservative bias).

Bryan predominantly selected Republican rulings for analysis (60%) with the highly-inflammatory Michele Bachmann and similarly noisy, opinionated pundits like Glenn Beck and Rush Limbaugh topping his list. His choice of Democrats follows a similar pattern of PolitiFact’s choices, with Barack Obama getting roughly one third of the Democrat ruling analyses.

Politi-Score: PolitiFact is not interested

PolitiFact *advertises* to Facebook commenters that it would appreciate any suggestions.  So I emailed the "Truthometer at PolitiFact" a few weeks ago, and asked them what they thought of the averages as I described in my previous post (although that is only the pundits--I have calculations on the non-pundits as well), and asked if there was any way they could be utilized.

It took about a week, but I received an email from Lou Jacobson with the following response: