Achievement and Quality: Higher Education in the Arts

Numbers and Evaluation in the Arts: Critical Questions

Boyle's Counting Paradoxes

Over-reliance on counting is a serious problem. It can lead to bad decision making and thus to all sorts of dysfunctions. For example, to an accounting firm, the purpose of business may appear to be to create numbers for accountants. But most, including accountants, see business fulfilling a broader range of functions. Accounting is just part of the whole, and at best, a service that helps the other parts function well.

Einstein is reported to have had the following statement on the wall in his study:  "Not everything that counts can be counted, and not everything that can be counted, counts."

In The Tyranny of Numbers (HarperCollins, 2000), British mathematician David Boyle outlines a number of counting paradoxes. We have listed these below and provided annotations with an arts example for each but the last.

1. You can count people, but you can't count individuals. For example, the relative success of all graduates of a program expressed in numbers does not predict what a specific individual has achieved or will achieve.
2. If you count the wrong thing, you go backwards. For example, if you reward high graduation rates, incentives for rigor at high levels are reduced.
3. Numbers replace trust, but make measuring even more untrustworthy. For example, any numbers that produce political, financial, or public relations disadvantages produce political challenges regarding the indicators chosen and the motives of those choosing them.
4. When numbers fail, we get more numbers. For example, when counting something doesn't seem to solve identified problems, there are calls to measure the thing more thoroughly or to find systems or people to measure it better. It is commonly said that the United States collects more education data than any other nation. Doing so does not seem to improve education. Nevertheless, most reform proposals call for more and better data collection.
5. The more we count, the less we understand. For example, when educational achievement is reduced to multiple choice test scores, significant information is lost about what a person or group really knows and is able to do, how well they can think and create with information, etc. This is why evaluation of educational achievement in the arts disciplines still relies so much on the audition or a portfolio of work to judge achievement and quality.
6. The more accurately we count, the more unreliable the figures. For example, quests for perfection in counting achievement can result in narrowing perspective to dysfunctional levels. We cannot tell much useful about the overall quality of a symphony performance by setting up measuring devices to count to high levels accuracy of the periodicity of all sixteenth notes played by the violins. Since we tend to count what can be counted easily, this paradox and its attendant unreliability is magnified in systems where there is over-reliance on counting.
7. The more we count, the less we can compare the figures. For example, there are tendencies to produce cause-and-effect arguments with sets of figures that have little relationship to each other, particularly with regard to the conditions that produced them. Comparing graduation rates in institutions with different missions and student demographics produces information of little value with regard to providing educational opportunity, maintaining the integrity of degree levels, or assuring the quality of teaching and learning in disciplinary content.
8. Measurements have a monstrous life of their own. For example, false numbers repeated as fact, or real numbers repeated out of context produce conditions for false analyses and wrong decisions.
9. When you count things, they get worse. That is, official statistics tend to get worse when society is worried about something. This may be due to better reporting due to heightened concern, or heightened concern may produce new criteria that raise the numbers, or that heightened concern produces calls for more numbers with all the consequences that we have mentioned in points 1 through 8.

More Information

• Questions
• Resources Regarding Numbers and Evaluation in the Arts
  

 _____________________________

Kindly report any Web site problems or broken links to Willa Shaffer (wshaffer@arts-accredit.org).

Thank you.