By Mark Wylie (written August 7)
Note: This is an Amazon.com review of the book "Turn Left: How Liberal Media Bias Distorts the American Mind" by Tim Groseclose. Mark's was one of fifteen reviews. Two of those fifteen reviews were by individuals who had received advance copies. Thirty-one of seventy-six people who read Mark's review found it "helpful." This is being posted as the second of a series directed at those who might view his book and research as definitive proof of liberal media bias.
In this book Tim Groseclose, a political scientist at UCLA, tackles the issue of supposed "liberal bias" in the news media. He reaches conclusions that are very provocative, and which will be warmly greeted by conservatives; however, his analysis is flawed, even crippled, by numerous, serious failings. His conclusions should be rejected.
Note: This is an Amazon.com review of the book "Turn Left: How Liberal Media Bias Distorts the American Mind" by Tim Groseclose. Mark's was one of fifteen reviews. Two of those fifteen reviews were by individuals who had received advance copies. Thirty-one of seventy-six people who read Mark's review found it "helpful." This is being posted as the second of a series directed at those who might view his book and research as definitive proof of liberal media bias.
In this book Tim Groseclose, a political scientist at UCLA, tackles the issue of supposed "liberal bias" in the news media. He reaches conclusions that are very provocative, and which will be warmly greeted by conservatives; however, his analysis is flawed, even crippled, by numerous, serious failings. His conclusions should be rejected.
The basic conclusions they reach are repeated in the book. Groseclose and Milyo find that almost every media outlet they studied has a "slant quotient" of well over 50, which they claim is a leftward slant (more on this issue below). In the book Groseclose goes further, and concludes that the "bias" of the media significantly impacts American elections, by swinging 8-10% of the vote to the Democrats; he claims that without media "bias," John McCain would have won in a landslide in 2008 with 56% of the vote.
So what's wrong here? Plenty. A full discussion of the shortcomings of this book would far exceed the word limit for reviews. There have been some significant online critiques of Groseclose's work, and I encourage readers who want to learn more to track down those written by Andrew Gelman, Brendan Nyhan and Paul Waldman, in particular. I'm going to focus on three basic criticisms:
1) One big problem with Groseclose's basic method concerns his use of the ADA voting ratings as a measure of ideology. Not only is he assuming that we can numerically measure political ideology at all by doing this, he is making the assumption that you can capture all major differences in ideology with a one-dimensional statistical measure.
To see why he's wrong let's do a little hypothetical exercise. Suppose we have two members of Congress, Ms. A and Mr. B. Ms. A is a consistent, principled libertarian; Mr. B is an equally principled communitarian. On economic issues, Ms. A almost never votes in favor of government intervention in the economy, while Mr. B is much more likely to vote in favor. Likewise, on social issues, Ms. A will vote to maximize individual freedom, while Mr. B is likely to vote in favor of many government limits on individual freedom. To a liberal group, like the ADA, or a conservative group, like the American Conservative Union, both Ms. A and Mr. B are going to be perceived as voting the "right" way about 50% of the time; they would tend to have voting scores that were about equal.
If you assume, as Groseclose does, that one-dimensional voting scores measure ideology with precise accuracy, you'd conclude from voting scores like ADA's that Ms. A and Mr. B are "about the same" ideologically. But, you'd be reaching this conclusion about two people who were on opposite sides of virtually all issues (at least in domestic policy). In other words, the conclusion that Ms. A and Mr. B are "the same" in their ideologies would be dead wrong. Trying to measure ideology on a one-dimensional scale loses a lot of information.
Many serious students of ideology recognize this problem, and use measures of ideology that have two or more dimensions. Readers may be familiar with the Political Compass or the Nolan Chart, which are fairly basic examples of this approach. Groseclose, however, doesn't seem to be aware of this serious shortcoming in his method.
2) Even if you believe that it's valid to measure ideology with a one-dimensional scale, there's a problem with Groseclose's method of estimating "political quotients" and "slant quotients." Recall how he determines the imputed ADA scores , or "PQ's," of think tanks and other political organizations--I described it above. When he does this, he makes a crucial assumption--that when a politician cites one of these organizations in a speech, they do so primarily because they are in broad agreement with the organization ideologically. Is that the case? A look at the imputed ADA ratings for political organizations suggests otherwise (to get these numbers you have to look up Groseclose's paper with Milyo online, they are omitted from the book).
One striking anomaly concerns the "PQ" of the American Civil Liberties Union. We all know that the ACLU is a pretty liberal outfit, right? So, you'd expect them to have a "political quotient" that's pretty high up on the scale--probably at least in the 70s, maybe the 80s. But when you look at Groseclose's numbers, he gives them a score of 48.9. That's right, his method says that the ACLU is a smidgen to the right of center politically.
How did this happen? Well, in his paper with Milyo, Groseclose lets slip the fact that a lot of the citations of the ACLU in his data set were made by Republicans. In fact, one-eighth of all citations of the ACLU were made by one man, Republican Senator Mitch McConnell of Kentucky. We can definitely state that Mitch McConnell was not citing the ACLU because he is a committed civil libertarian; the ACLU does their own ratings of voting records, and Sen. McConnell gets consistently low scores from them (a 14% in 2008, for instance).
Another anomaly can be seen in the "PQ's" of organizations involved in the gun control debate. Let me propose another hypothetical. Suppose we had a news outlet that, in covering gun control, gave equal weight to the views of 1) Handgun Control (liberal, pro-gun control) and 2) the National Rifle Association (conservative, anti-gun control). We'd all agree that, other things being equal, that news outlet was giving balanced coverage to that issue.
However, Groseclose's method of analysis would interpret that balanced coverage as "slanted" to the left. The reason is that, while Handgun Control has a "PQ" of 77.3 (pretty liberal, and about what I'd expect), the NRA has a "PQ" of 45.9 (you'd expect them to have a score in the 20s). So the balanced coverage of our hypothetical news outlet would have a "slant quotient" in the low 60s on the gun control issue, which Groseclose would interpret as "liberal bias."
Why do these weird results come up? Because Groseclose's core assumption--that citations of organizations are driven by ideology--is not valid. For instance, the ACLU's score is driven by their being cited by many Republicans in the debate over a single law, the McCain-Feingold bill. Because the ACLU, a liberal organization, was opposed to a bill supported by most liberals, right-wing Republicans tried to score debating points by citing them. I strongly suspect that the NRA's score is similarly due to liberals citing them in gun control debates. The examples of the ACLU and the NRA, and some others I don't have time to go into, make it clear that many politicians cite think tanks and other organizations not because of ideological agreement, but for tactical reasons.
3) Even if you reject my criticisms of Groseclose's basic method--his model, in economists' language--there is still a very big problem with his statistical analysis. He has a very severe example of what is known as a sensitivity problem.
So what's that? Statistical research has a sensitivity problem if making a change in one element of the data you use or the parameters of your analysis (think of these as the "inputs") results in a significant change in your conclusions (the "output"). For example, if I did a statistical analysis that used data from all 50 states and reached conclusion X, but dropping one or two states out would reverse that conclusion or seriously modify it, my analysis would have a sensitivity problem. Good social scientists do not base sweeping conclusions on analysis with such a problem.
Groseclose's work has a number of sensitivity problems. Some of them have been found by a political scientist at Carnegie Mellon University named John Gasper, who has a paper critiquing Groseclose's work coming out--it's accessible online. Gasper finds that there are several of these sensitivity problems. His most striking finding is that if you remove the citations of a single organization from the analysis (the National Taxpayer's Union), the average "slant quotient" of media outlets drops by 9 points.
What would that mean for Groseclose's conclusions. First, instead of most media outlets having a slant to the left, most outlets--14 out of 20--would have a "slant quotient" of somewhere between 45 and 57, which puts them right around the center of the political spectrum (Groseclose has determined that the center is at a "PQ" of just over 50). Moreover, if you apply that 9 point drop to Groseclose's overall measure of media "slant," which he calls the "media mu," you get a "mu" of just under 50, which would mean that the media were barely slanted at all, and that the slant that did exist was slightly to the right. If that were the case, Groseclose's conclusion about the media's effect on the 2008 election goes out the window entirely.
Let me reiterate what Gasper's paper is saying: if you redo Groseclose's work, using all the same data and doing all the number-crunching just the same, but you drop out the media citations of the National Taxpayer's Union while making no other changes at all, all of Groseclose's big conclusions are invalidated.
Another source of a sensitivity problem is the way Groseclose defines the "center" of the electorate. He makes the assumption that the center of the electorate, ideologically, will be exactly the same as the center of Congress (a debatable assumption itself). To find the center of Congress, he then calculates the mean ADA score for all members of Congress, which he finds is 50.4. The problem--anyone who's taken a stats course has learned this--is that you don't use the mean to find the statistical center, you use the median.
So what's the median ADA score for Congress. Lacking the time and resources to do a massive study like Groseclose`s, I did a quick, back-of-an-envelope estimate using data in the latest Almanac of American Politics. I found that the median ADA score for Congress, taking the members of each House who were the closest to the center, was about a 70.
If the center of the electorate is a ADA score/"political quotient" of 70 instead of 50, once again, all of Groseclose's conclusions are overturned. Even if the median ADA score turns out to be a 65 or a 60, that is still the case.
The point here is not to say "My numbers or Gasper's are right, and Groseclose's are wrong." Rather, the point is that Groseclose's numbers, and his conclusions, depend very strongly on some specific decisions he made about what data to include in his analysis and what values to give to certain key variables. Making very small and very reasonable changes to those decisions radically changes the conclusions that can be drawn. Therefore, Groseclose's conclusions don't really have the rigorous statistical foundation that he claims for them.
I apologize for the length of this review. I wanted to do more than just express my opinion that Groseclose is wrong in his conclusions; I wanted to explain why I have concluded that he's wrong. This required me to go into quite a bit more depth than is normal for an Amazon review.
I've rated this book at one star. I thought about this issue quite a bit. If I were evaluating this book as a conservative polemic (like the Ben Shapiro book I reviewed a few weeks back), I'd have given it a higher rating. I do give him credit for trying to study his subject in a systematic way, something that conservative polemicists almost never attempt. However, it's clear from reading the book, and from the plugs given to the book by high-powered economists like Steve Levitt, that Groseclose wants this book to be taken as a serious work of empirical social science research. That being the case, my standards rise quite a bit, and my rating of the book drops accordingly.
Postscript: Click here for part 3, which is the initial criticism of Groseclose's book by Paul Waldman.
If you assume, as Groseclose does, that one-dimensional voting scores measure ideology with precise accuracy, you'd conclude from voting scores like ADA's that Ms. A and Mr. B are "about the same" ideologically. But, you'd be reaching this conclusion about two people who were on opposite sides of virtually all issues (at least in domestic policy). In other words, the conclusion that Ms. A and Mr. B are "the same" in their ideologies would be dead wrong. Trying to measure ideology on a one-dimensional scale loses a lot of information.
Many serious students of ideology recognize this problem, and use measures of ideology that have two or more dimensions. Readers may be familiar with the Political Compass or the Nolan Chart, which are fairly basic examples of this approach. Groseclose, however, doesn't seem to be aware of this serious shortcoming in his method.
2) Even if you believe that it's valid to measure ideology with a one-dimensional scale, there's a problem with Groseclose's method of estimating "political quotients" and "slant quotients." Recall how he determines the imputed ADA scores , or "PQ's," of think tanks and other political organizations--I described it above. When he does this, he makes a crucial assumption--that when a politician cites one of these organizations in a speech, they do so primarily because they are in broad agreement with the organization ideologically. Is that the case? A look at the imputed ADA ratings for political organizations suggests otherwise (to get these numbers you have to look up Groseclose's paper with Milyo online, they are omitted from the book).
One striking anomaly concerns the "PQ" of the American Civil Liberties Union. We all know that the ACLU is a pretty liberal outfit, right? So, you'd expect them to have a "political quotient" that's pretty high up on the scale--probably at least in the 70s, maybe the 80s. But when you look at Groseclose's numbers, he gives them a score of 48.9. That's right, his method says that the ACLU is a smidgen to the right of center politically.
How did this happen? Well, in his paper with Milyo, Groseclose lets slip the fact that a lot of the citations of the ACLU in his data set were made by Republicans. In fact, one-eighth of all citations of the ACLU were made by one man, Republican Senator Mitch McConnell of Kentucky. We can definitely state that Mitch McConnell was not citing the ACLU because he is a committed civil libertarian; the ACLU does their own ratings of voting records, and Sen. McConnell gets consistently low scores from them (a 14% in 2008, for instance).
Another anomaly can be seen in the "PQ's" of organizations involved in the gun control debate. Let me propose another hypothetical. Suppose we had a news outlet that, in covering gun control, gave equal weight to the views of 1) Handgun Control (liberal, pro-gun control) and 2) the National Rifle Association (conservative, anti-gun control). We'd all agree that, other things being equal, that news outlet was giving balanced coverage to that issue.
However, Groseclose's method of analysis would interpret that balanced coverage as "slanted" to the left. The reason is that, while Handgun Control has a "PQ" of 77.3 (pretty liberal, and about what I'd expect), the NRA has a "PQ" of 45.9 (you'd expect them to have a score in the 20s). So the balanced coverage of our hypothetical news outlet would have a "slant quotient" in the low 60s on the gun control issue, which Groseclose would interpret as "liberal bias."
Why do these weird results come up? Because Groseclose's core assumption--that citations of organizations are driven by ideology--is not valid. For instance, the ACLU's score is driven by their being cited by many Republicans in the debate over a single law, the McCain-Feingold bill. Because the ACLU, a liberal organization, was opposed to a bill supported by most liberals, right-wing Republicans tried to score debating points by citing them. I strongly suspect that the NRA's score is similarly due to liberals citing them in gun control debates. The examples of the ACLU and the NRA, and some others I don't have time to go into, make it clear that many politicians cite think tanks and other organizations not because of ideological agreement, but for tactical reasons.
3) Even if you reject my criticisms of Groseclose's basic method--his model, in economists' language--there is still a very big problem with his statistical analysis. He has a very severe example of what is known as a sensitivity problem.
So what's that? Statistical research has a sensitivity problem if making a change in one element of the data you use or the parameters of your analysis (think of these as the "inputs") results in a significant change in your conclusions (the "output"). For example, if I did a statistical analysis that used data from all 50 states and reached conclusion X, but dropping one or two states out would reverse that conclusion or seriously modify it, my analysis would have a sensitivity problem. Good social scientists do not base sweeping conclusions on analysis with such a problem.
Groseclose's work has a number of sensitivity problems. Some of them have been found by a political scientist at Carnegie Mellon University named John Gasper, who has a paper critiquing Groseclose's work coming out--it's accessible online. Gasper finds that there are several of these sensitivity problems. His most striking finding is that if you remove the citations of a single organization from the analysis (the National Taxpayer's Union), the average "slant quotient" of media outlets drops by 9 points.
What would that mean for Groseclose's conclusions. First, instead of most media outlets having a slant to the left, most outlets--14 out of 20--would have a "slant quotient" of somewhere between 45 and 57, which puts them right around the center of the political spectrum (Groseclose has determined that the center is at a "PQ" of just over 50). Moreover, if you apply that 9 point drop to Groseclose's overall measure of media "slant," which he calls the "media mu," you get a "mu" of just under 50, which would mean that the media were barely slanted at all, and that the slant that did exist was slightly to the right. If that were the case, Groseclose's conclusion about the media's effect on the 2008 election goes out the window entirely.
Let me reiterate what Gasper's paper is saying: if you redo Groseclose's work, using all the same data and doing all the number-crunching just the same, but you drop out the media citations of the National Taxpayer's Union while making no other changes at all, all of Groseclose's big conclusions are invalidated.
Another source of a sensitivity problem is the way Groseclose defines the "center" of the electorate. He makes the assumption that the center of the electorate, ideologically, will be exactly the same as the center of Congress (a debatable assumption itself). To find the center of Congress, he then calculates the mean ADA score for all members of Congress, which he finds is 50.4. The problem--anyone who's taken a stats course has learned this--is that you don't use the mean to find the statistical center, you use the median.
So what's the median ADA score for Congress. Lacking the time and resources to do a massive study like Groseclose`s, I did a quick, back-of-an-envelope estimate using data in the latest Almanac of American Politics. I found that the median ADA score for Congress, taking the members of each House who were the closest to the center, was about a 70.
If the center of the electorate is a ADA score/"political quotient" of 70 instead of 50, once again, all of Groseclose's conclusions are overturned. Even if the median ADA score turns out to be a 65 or a 60, that is still the case.
The point here is not to say "My numbers or Gasper's are right, and Groseclose's are wrong." Rather, the point is that Groseclose's numbers, and his conclusions, depend very strongly on some specific decisions he made about what data to include in his analysis and what values to give to certain key variables. Making very small and very reasonable changes to those decisions radically changes the conclusions that can be drawn. Therefore, Groseclose's conclusions don't really have the rigorous statistical foundation that he claims for them.
I apologize for the length of this review. I wanted to do more than just express my opinion that Groseclose is wrong in his conclusions; I wanted to explain why I have concluded that he's wrong. This required me to go into quite a bit more depth than is normal for an Amazon review.
I've rated this book at one star. I thought about this issue quite a bit. If I were evaluating this book as a conservative polemic (like the Ben Shapiro book I reviewed a few weeks back), I'd have given it a higher rating. I do give him credit for trying to study his subject in a systematic way, something that conservative polemicists almost never attempt. However, it's clear from reading the book, and from the plugs given to the book by high-powered economists like Steve Levitt, that Groseclose wants this book to be taken as a serious work of empirical social science research. That being the case, my standards rise quite a bit, and my rating of the book drops accordingly.
Postscript: Click here for part 3, which is the initial criticism of Groseclose's book by Paul Waldman.
No comments:
Post a Comment