Quantitative Information in Knowledge Claims

Return to Critical Thinking Reference

As we saw in discussing knowledge claims, one powerful direction that the growth of knowledge has taken has been to quantify our experience. To learn more about quantificational methods in knowledge claims, we begin with the concept of measurement itself. We consider ways that measurements can be misunderstood or used deceptively. People have difficulty, for example, in comparing measurements. We'll look at specific problems that arise from the use of percentages, linear and non-linear relationships, and surveys. Another source of difficulty in the use of quantitative information in knowledge claims comes from our incomplete grasp of probability. So we take some time to discuss some basic ideas in probability and some ways that people makes mistakes in thinking about knowledge by misunderstanding probability. As always, our focus is on using conceptual knowledge about these topics to find practical advice about how to think well.

Surveys

When most people think of surveys, they think of public opinion polls such as those produced by respected polling companies, such as the Gallup Organization (www.gallup.com). Because opinion polls enter into reflective discussions so often, we will focus most of our attention on them. But some of what we will say about good opinion surveying applies as well to any situation in which you want to sample the frequency of some event or property, whether you manufacturing process, or the state of the rush hour traffic jam.

Whether you are studying people's opinions or sampling some other kind of system, the central concept of good surveys is representative sampling. In what is still the textbook example of unrepresentative sampling, the 1936 Literary Digest poll predicted that Alf Landon would beat Franklin Delano Roosevelt in the presidential election. The sample population, taken from telephone lists and auto registration lists, was over 2 million people, far more than is needed with current polling techniques. The U.S. population can usually be surveyed accurately by contacting about 1200 people. The prediction was false because the sample contained an unrepresentative number of wealthy Americans (at that time, people who owned cars or telephones) with a class interest in voting for Landon. In other words, wealthy Americans had a far greater chance of appearing in the sample population than in the total population. The general difference in economic status between the sample population and the general population was a relevant difference because it made a difference in how people from each group felt about the presidential race.

The key to avoiding this kind of problem is to make sure that the sample surveyed is representative of the whole group. A sample is representative if every relevant difference in the sample has an equal chance of appearing in the total population. What's a relevant difference? A relevant difference is one which would affect the opinion being measured. In the example of the presidential poll, owning a car or telephone is a relevant difference because that predicts class affiliation, which in turn predicts attitudes and views that divide people voting for president. Notice that a relevant difference isn't always something directly related to the issue being polled. When reading a poll result, you should be able to learn how many people were questioned, over what period of time, and by what means.

Professional pollsters are usually able to make survey samples representative, but the next area of bias is harder to control. In a variety of ways, the questions asked in a poll can skew results. On very controversial topics, the wording of the question can determine whether a majority comes out in favor of the question or against it. For example, on the still controversial topic of gay marriage, the way a question is phrased can determine whether a majority supports or opposes it. For this reason, when reporting a poll result you should always report the actual question asked.

Sometimes a poll result depends upon how many choices of response are provided, how options are lumped together, whether terms are used that are "loaded" or simply perceived differently by the respondents to the poll. In one case, a poll on joblessness used the work "laid off," as in, "Have you been laid off from your job within the last six months?" A "lay off typically refers to a temporary loss of a job. For some reason, a substantial percentage of the respondents were later found to have understood the phrase to mean "fired." In other cases, polling experts have found that people systematically under-report some kinds of information (criminal behavior, of course, unflattering information about sexual experience, but also less sensitive things such as the number of visits to a doctor). People tend to over report voting behavior (saying they have voted when in fact they haven't). This is an important source of bias to control since people who do not vote tend to have different political views than people who do vote. If your goal is to predict the outcome of an election. you need to be able to sample likely voters.

In general, pollsters try to control for these kinds of measurement errors by prompting poll subjects in different ways. Sometimes a series of questions are asked to help people remember their past behavior. With controversial terms or words that are not understood consistently, a pollster may simply replace the term by a phrase that supplies its meaning. For example, people have different ways of understanding a word like "few" in phrases like "over the last few years," so a good pollster will probably give a set of choices specified with specific year ranges.

More subtle measurement errors occur when people are polled about things they do not understand or when the answer to one question affects the answer to the next. Polls have been done using fictitious geographical or ethnic information and almost everyone asked seems to be able to answer questions about this fictional content. In one famous example from the last century, respondents gave opinions about three nationalities that do not exist: Danireans, Pireneans, and Wallonians. Offering options such as "I do not have enough information" or "undecided" can help respondents acknowledge their lack of understanding.

A famous study from the 1950's, during the height of the anti-communist "red scare" in the U.S., illustrates the measurement error that changing the order of questions can introduce. Only 36% of respondents said "yes" when asked, "Do you think the United States should let Communist reporters from other countries come in here and send back to their papers the news as they see it?" However, when the question was preceded by a question about free reporting on Russia by U.S. reporters, 73% agreed to it. Ideally, then, you should be able to learn not only the specific questions asked, but also the full set of questions and the order in which they were asked.

Often we want good polling data about very current topics that are unfolding in the news. Even with careful sampling and well written questions. poll results on current events can reflect the volatility of the public's opinions as they are swayed to and fro by current events. At the time of this writing. state courts in Massachusetts and elsewhere are issuing rulings on gay marriage. There is talk about a constitutional amendment to prohibit gay marriages. Some municipalities have been issuing marriage licenses to gay couples. Historically, and especially prior to the recent controversies over the issue, strong majorities of the public have opposed gay marriage and many polls continue to reflect that. However here are three recent poll result that paint a complex and inconsistent picture:

"Poll: Most Oppose Gay Marriage"

In a CBS News poll conducted immediately after President Bush endorsed a constitutional ban on gay marriage, 59% of Americans said they would favor an amendment to the Constitution that would "allow marriage only between a man and a woman," up slightly from 55% last December.(CBSNEWS.Com, "Poll: Most Oppose Gay Weddings," New York, Feb 28, 2004, viewed April 20, 2004, http://www.cbsnews.com/stories/2004/02/24/national/main601828.shtml.)

"Poll Shows Mass. Gay Marriage Ruling OK"

Both polls released Sunday found opposition to the proposed constitutional amendment — 53% opposed and 36% in favor in the Globe/WBZ poll of 400 Massachusetts resident, and 54% opposed and 36% in favor in the Herald poll of 405 residents.

Another poll, by Merrimack College, found that 75% of Massachusetts adults support either allowing gay marriage or civil unions. That poll of 491 adults was conducted in the days before and after the decision, but the numbers didn't shift after the ruling. The margin of sampling error was plus or minus 5 percentage points. (18 viewed April 20, 2004, http://www.usatoday.com/news/nation/2003-ll-23-gaymarriage-poll_x.htm.)

These two quotes, from news sources several months apart, illustrate the volatility of poll results. Strong majorities of the public opposed gay marriage prior to George Bush's proposed constitutional amendment, and the poll following his proposal shows a 4 percentage point boost. Did the authority of the president's recommendation influence public opinion? The later polls followed an historic November 18, 2003 decision by the Massachusetts Supreme Court that gay couples have a right to marry. The results suggest that public opinion in Massachusetts is not typical of the nation as a whole (though to know if this were a trend you would want to more data on where Massachusetts residents stood on the earlier poll. Were Massachusetts adults influenced by the authority of their own Supreme Court? Is there a long term trend toward smaller majorities opposing gay marriage? e? These are questions that might be easier to answer in a year or so by searching for poll results that place the late 2003 and early 2004 polls in perspective.

Sometimes volatility in polling results is a result of social change and fluid public opinion, as in the example above. But survey data can be influence by public trends in other ways as well. Leaving opinion polling for the moment, when attention is turned to medical or social problems that had not previously been discussed, one often reads about shocking "increases" in the incidence of these problems. For example, as eating disorders began to be recognized as real medical and mental health problems, women started discussing this problem with their physicians. If you watched statistics on the incidence of eating disorders you might think there was a surge in new cases of the illness when, in fact, it could be that there was no increase in the illness, just an increase in the willingness to report it. Of course, this would not make the problem any less serious.

Another, but by no means the last, technical problem to consider in public opinion polling is "margin of error." While the computation of a margin of error is a technical question beyond our concern, the important point is that good quality polls report their "margin of error" or "sampling error," usually in terms of "plus or minus x percentage points. Notice that some of the polls quoted above report their margin of error and others do not.

Having looked at the problem of representative sampling, the wording of the polls themselves, and the various ways that poll results can be skewed, you might be wondering if surveys and opinion polls are really worth very much. In spite of all of these ways they can fail, the answer is a definite "yes." The proof of the value of modem surveying techniques is in their predictive power. Polls can reliably predict the outcomes of elections when the results are not closer than the margin of error of the poll. Surveys of other phenomena can give us an accurate look at conditions and events that are too numerous to count exhaustively. As long as we are aware of the hallmarks of professional polling techniques, we are unlikely to be fooled by the results.

Of course, having a reliable poll result may only get you to the beginning of the interpretive reflective process. Polls are more or less relevant for certain questions. Knowing how many people are experiencing eating disorders doesn't tell you what causes them, but it can help you reason about their causes. Inductive inference is, after all, about reasoning from patterns in our experience and polling and surveying is a basic activity in looking for such patterns.