Politi-Score: Insidious Effects

I devote a lot of my blog to someone I often term “my conservative counterpart” because he seems to have his life devoted to “exposing PolitiFact” for its supposed liberal bias at his weblogs Sublime Bloviations and PolitiFact Bias. When he (Bryan White) first became aware of my counterpoint blog, he immediately wrote some pieces trashing me at another shadow-blog he had called “Bad Blogs' Blood.” (If you Google “politi-psychotics” you will see his blog in the results, referring to my “bad reporting” and “bad thinking” and “publishing predominantly a load of nonsense.” I must have had him worried!) After he wrote me four consecutive letters one day on Facebook which I chose to ignore, he in turn ignored me, by making a point to never mention my blog on his main blog Sublime Bloviations.  Recently, however, he’s made comments (as he does every now and then) where it’s obvious the message is directed to me, as in this case, since he is very much aware of the Politi-Score:

Afters: The insidious effects of the type of selection bias found in this PolitiFact story have their full effect when PolitiFact encourages readers to look at the collected statements about a person in order to develop an impression about the person. Using Eisnaugle as an example, the reader of Eisnaugle's PolitiFact "report card" (they actually call it that!) would see a lone "False" rating. Yet if PolitiFact had chosen either of the other two statements to rate, Eisnaugle would have a lone "True" rating. Or if PolitiFact had rated all three, Eisnaugle could expect to have two "True" ratings and one "False" rating.

What good is a grade point average created under these types of conditions?

Short answer: There is no “good” to a grade point average (GPA or Politi-Score) under these types of conditions. It is an irrelevant measure.

Long Answer: In order to get a reasonably “good” GPA (Politi-Score), there has to be a sufficiently large quantity of rulings over a sustained period of time. Probably the only politicians who meet this measure, as of the end of 2010, are Barack Obama and John McCain, with over 400 rulings between them.

As I state on my “About Page” which shows how the “Politi-Score” is calculated, (italics added)

…. In fact, roughly a third of the total rulings are from individuals/groups who have not had more than a single statement ruled on by PolitiFact. Due to this, most of my analyses is of the population, say, of Democrats, Republicans, or a PolitiFact state.

Even when doing a PolitiFact state-by-state analysis, this sampling factor is taken into account: (italics added)

…And if the Politi-Score is any evidence, that back and forth just managed a "Barely True", far less than the general average. Yes, among the other states, PF Rhode Island may have a Politi-Score of 27, very close to that of PF Wisconsin, but they only had 22 rulings, not enough for a good measure…

The last Politi-Score on individuals was done earlier this year for ALL PolitiFact rulings since it began through the end of 2010, and only for those who had more than 10 rulings. In some respects, even 10 rulings may not be enough. But to show that PolitiFact has been quite extensive or “comprehensive” in its statement selection by individual (as opposed to focusing on a selection of certain politicians), if you were to score only those who had rulings on 20 or more statements, there’d be only four Democrats and 11 Republicans on the list (see below).

The last Politi-Score on individuals were those with 10 or more rulings

At first, I did the individual rulings for the fun of it, and that’s where it will probably remain due to the too-small-sample proviso (or PolitiFact “goes out of business.”). In the meantime I am collecting other data in order to make the scores more meaningful, such as a recent review of PolitiFact National writers and PolitiFact subjects using the application of the Politi-Score.

At first I was very excited about the individual Politi-Score because it appeared to be an easier, over-all way that an individual’s PolitiFact ruling score card could be relatively evaluated, compared to just looking at the number of True, Mostly True, etc. I actually submitted it as a suggestion to PolitiFact, and Louis Jacobson was kind enough to reply: “Our general orientation is not to make these calculations. The analogy is that a baseball umpire doesn't calculate his safe/out ratio or ball/strike ratio for different teams.” Maybe he’s right.  But it might work in other ways.