Beginner's Guide to Efficiency Measurement
SLMQ Volume 22, Number 2, Winter 1994
Sue Easun, Assistant Professor, Faculty of Library and Information Studies, University of Toronto.
We in the library and information profession have yet to figure out what to do with the topic of efficiency measurement; it remains in large part the privileged property of engineers and economists. Yet there isn't one among us who hasn't at some point been faced with the need to act efficiently, to make the best of a given situation. In fact, thanks to the current economic climate, we are likely to find that the so-called "givens" of these situations are fewer than ever before.
Why, then, are we so hesitant to evaluate ourselves in terms of our efficiency? One possible reason is that efficiency experts like engineers and economists are willing to manipulate numbers until they come up with "optimal" combinations of inputs and outputs, while the rest of us are usually satisfied with doing well enough to get by. Another is that people concerned with efficiency are often focused on managerial behavior, while we, of necessity, have been far more focused on securing our position within our respective schools. A third is that professionals dealing with human resources do not take kindly to strategies that equate professional performance with that of inanimate objects.
Nonetheless, the need to act efficiently in a time of limited resources is unarguable. Even at the best of times, budget constraints ensure that there is only so much money available for deployment; and when we must choose among competing options, whether or not there is a price tag attached, we too are constrained to behave efficiently. These points in turn lead to the following set of questions:
- What do we mean by "efficient behavior"?
- What is efficient behavior in library and information settings?
- How can we measure it?
What Is Efficient Behavior?Why does anyone want to be efficient? We might begin by saying that people choose to be efficient because they want to conserve energy of some sort. For instance:
- A shortcut or two on the way to the office saves time and gas.
- Turning out the lights when leaving a room saves electricity.
- Using a word processor saves time, money, and aggravation... or does it?
We are then faced with the question: Do computers make us efficient, inefficient, or a little bit of both? On the surface the other examples lacked this element of doubt: taking shortcuts and turning out lights certainly seem to be good ideas. But what if you aren't the only person using that shortcut and you find yourself caught in a traffic jam? What if, because of that shortcut, you decide not to join a carpool? Are you still saving time and gas? Could you save even more by considering and acting upon other alternatives?
It would seem, then, that efficiency is not an absolute standard. Rather, it is a relative concept that invites comparison. If I consider myself efficient, I must still ask, "Compared to whom (or what)?" Am I more efficient than my colleagues? Am I more efficient than I was last year at this time? Or what if I'm efficient at using a word processor but not at meeting deadlines: Does this automatically make me inefficient? And if so, what would it take to make me efficient again?
These are not trivial questions. While it may be that most of us don't make such questioning an integral part of our lives, our actions often serve as answers. It is not unheard of for someone to say, "I know the fastest way to get there" or "Eating oat bran is the best way to reduce cholesterol." Such statements imply that these people have taken the time to test different routes and different types of cereal before making their respective assessments. It is important, however, to realize that neither mutes nor cereals are efficient in and of themselves; they can only be used efficiently, depending on whether the person in question wants to get the most out of his or her investment as possible or to invest less and maintain a status quo.
Even so, there are still limits to be considered. A car can go only so fast. One can eat only so much oat bran. By the same token, a car must travel at some speed if it is to move at all, and at least some oat bran must be eaten before it can take effect. Clearly, "fastest" and "best" are points on a continuum.
These examples illustrate four important concepts that any and all definitions of efficiency must accommodate:
- Efficiency is relative. One acts efficiently when one is able to show that a particular course of action has a greater potential payoff than another.
- Efficiency is conditional. One must establish not only the criteria by which one's actions are to be judged but also the factors that contribute to efficiency in a given context.
- Efficiency is a measure of performance, not an inherent quality. While it may indicate that we have done the job right, it in no way ensures that we have done the right job.
- Efficiency is constrained. There are always limits, both to what one has and to what one can do with it.
What Is Efficient Behavior in Library and Information Settings?In order to answer this question, we must first test three assumptions. The first is that we recognize ourselves as resource managers. The second is that we really do want our libraries and information centers to be productive-that our job is to encourage people to use our resources and that we want them to have a satisfactory experience so that they'll come back and use them some more. The third assumption is that, in order to become more productive, we Beer to attract more resources, presumably so that we can provide more and better services, which in turn should attract more people.
The problem underlying assumptions two and three is that they are based on the premise that "more is better"-a criterion more appropriate to selling oat bran than to selling ideas. Similarly, when we talk about wanting to increase use and user satisfaction, we would be hard-pressed to assign to these any kind of market exchange value. On the contrary, what we're looking for is some indication that we have correctly anticipated the "market demand" for the information our goods and services make available. Thus, although we may be rewarded if our users are satisfied-anything from a gracious thank you to a generous donation-there is no price tag attached to signal that all parties involved share a common system of values.
Under such circumstances, then, it is dangerous to assume that "more" resources are going to enable us to anticipate "better," because we have no way of determining when the limits of "more" and "better" have been reached. Nonetheless, library and information center budgets (and human nature) are such that we never really feel we have enough money to produce as much as we'd like. Somewhere along the line, we need to make one of the following trade-off's:
- We can try to stretch our budget dollars to provide more resources, so as to increase our level of output.
- We can try to extend the range of uses being made of the library, so as to increase our mix of outputs.
- We can try for a little of both.
Whether our problem is too many books that aren't being used or not enough certified staff to provide instructional support, assessing efficiency can help us understand how we might accomplish more with less. Because libraries and information centers are not profit-making organizations, however, it is a mistake to assume that resources can be combined in such a way as to guarantee some particular level of "profitable" use. Nor can we be certain that use is automatically profitable by virtue of its having been observed and recorded. Instead, it seems that our traditional measures of library use -circulation, questions answered, units taught, and the like-are more likely to serve as measures of user interest, or demand, than as measures of productivity, or supply. In other words, use of library and information resources is dependent not only on what (and how much) of any par-titular resource has been provided but on whether users stand to gain something they value from having used it.
Libraries and information centers, then, are productive not only because they are used but because the uses made of them are valued. Just as wheat, when it takes the form of bread, is accorded a higher value than when it takes the form of flour, so do library and information resources increase in value when the uses made of them serve a particular purpose. But while producers and consumers alike may agree that bread is deserving of its relatively higher value, the value placed on an information resource will be highly dependent upon the various perceptions of those making the assessment. This variability in turn raises the question of whether the value added to such a resource through the provision and use of a particular service(s) is the same value users believe they have received and, further, whether it is the value its provider originally had in mind.
The application of microeconomics production theory to the study of libraries and information centers focuses on the allocation of resources. It asks whether, given a fixed pool of money, a library could increase its output by changing a particular mix of inputs. Likewise, the allocation process itself can be monitored through the distribution (or flow) of resources through various levels of decision making. However, it is not enough to ask whether a given mix of inputs is capable of producing a maximum amount of output, for there is still the question of whether there is another mix of outputs of potentially greater value.
All libraries and information centers face the task of al locating scarce resources among competing ends. However, an efficient allocation of resources means neither that the library in question has met some target goal nor that its resources have been well utilized in attaining that goal; it means simply that scarce resources have not been wasted. Thus, whether the library's goals would be better served by a different level or mix of outputs, or how best to ensure that other libraries produce comparable quantifies of these outputs, is not the central issue.
At the same time, if we are to ensure that scarce resources are not being wasted, we must pay some attention to whether they are being used effectively. What is called for, in turn, is research that looks at both effectiveness and efficiency and-here's the crucial point-distinguishes between them. On the one hand, an effective library will accomplish what it sets out to accomplish: There is an implied relationship between the library's objectives and the mix of outputs it decides it should produce. On the other hand, an efficient library will use the resources at its disposal to produce as much of those outputs as it can at as little cost as possible. And while it is certainly possible for a library to be effective but not efficient (and vice versa), we may still wonder whether effective libraries are more efficient, or efficient libraries more effective.
How Do We Measure Efficiency?Given that we are all convinced that efficient behavior is an essential component of any functioning system and that its achievement is not an absolute but a matter of degree, we can now turn our attention to the question of how best to assess it.
First, we need to determine whether potential is being wasted, whether it be the potential to increase the amount of output produced and/or the potential to decrease the amount of input required. Such waste can take one of three forms. It can result from the underemployment of available inputs, as when the library is not taking advantage of the resources it currently possesses. It can result from technical inefficiencies inherent in the production process, as when the library is not taking advantage of how these resources might be combined for greater effect. Or it can arise from imperfections in the structure of the organization itself, as when administrative practices prevent the pursuit of an optimal course of action.
Second, we need a standard against which to compare ourselves. Any given library will to some extent provide the same services and types of materials as other libraries involved in the same line of work, but each will have its own idiosyncrasies in procedure, policy, and organizational climate. Thus, while the desired end may be the same, the means to that end is likely to differ from library to library.
Third, we need to identify some incentive for efficient behavior. Certainly all libraries wish to ensure their continued existence, and all are concerned with resource scarcity. Further, it seems reasonable to expect that survival will be dependent (at least in part) on the availability of certain requisite resources and the ability of decision makers to convert these into desired outputs. At the same time, no two libraries have exactly the same resources with which to work and so will not necessarily provide the same mix of services; additionally, different services will be used at different times by different people. Nonetheless, given the current fiscal climate, no public institution can afford waste, whether it wishes to recognize it or not. Libraries and information centers must be prepared to face pressure to justify current expenditures, cope with possible reductions, and/or support requests for increases-all good reasons to behave efficiently.
Finally, we need in some way to incorporate the concept of "value added" in order to account for differing perceptions of the value of a particular service or resource to our different constituencies-users, administrators, and professional staff.
So how will all this happen? For an answer, I suggest we leave our professional world for a moment and go shoe shopping.
Let us begin by assuming that all of us do indeed want to shop for shoes. And it's probably a good idea if each of us has some idea of what we want to buy: a shoe that fits, a certain color, an affordable price range, one or more pairs, etc. Probably we will have to find our perfect shoe the hard way: by trying on pair after pair and simultaneously trying the patience of the sales staff. But let's suppose that we could simply approach a salesperson with our lengthy list of requirements, that he would agree to match that list against his stock and come up with the shoes we would have bought anyway, and that each of us leaves the store with at least one pair of shoes.
Now suppose we get together for lunch a couple of months later and reminisce about all the fun we had on our shopping spree. Inevitably, someone will wonder aloud who got the best deal. As a group, how will we figure it out?
Obviously, trying on each other's shoes won't work. So perhaps we decide to find another salesperson, present her with all our shoes, and ask her as an expert to make the decision for us. Poor salesperson!
In the hopes of getting rid of us, she might slip into the stockroom and emerge with what she believes is the quintessential shoe, against which she says we can compare our purchases. But what one shoe can be all sizes, all colors, and all prices to all people? What we really need her to do is to take all the criteria we used in selecting our shoes (size, price, quantity, color); plug in some additional measures (how many times we've worn them, whether they've had to be repaired, whether we still like them); and consider all these factors simultaneously for everyone in the group. The winners, so to speak, would be those of us who attained the 'best balance of all of the above.
Now let's go back to libraries, specifically school library media centers. In our example, we assumed that wanting to buy a pair of shoes was reason enough to do so; for library media centers, however, it's not quite that easy. The most obvious difference is that the library media center may find that the staff and materials it requires are completely out of its price range. Thus, what it needs and what it can afford may be totally irreconcilable. Further, library media center personnel are not apt to have a great deal of control over many of the decisions that have to be made, because so much depends on the current level of funding, the organizational climate of the school, and the personal inclinations of various layers of administrators.
But other aspects of our example do hold. First, money for school library media centers is tight, so it is not difficult to imagine a group of library media specialists getting together and wondering who among them was getting a better deal. Second, they obviously can't all exchange facilities to see if they prefer someone else's setup, so they too may decide to consult with an expert-but this time the expert is not an omniscient salesperson but a mathematical model.
Now a new question arises: What sort of model is capable of identifying inefficiencies in our particular kind of resource management?
Data Envelopment Analysis and the Concept of Efficient ChoiceModeling, in the generic sense, is an alternative decision-making process in which the model functions as an abstract and simplified version of the real world. Its value for libraries and other systems-based organizations lies in its ability to bring together disparate pieces of information and test whether they are consistent with each other. The addition of mathematics serves merely to move both simplification and abstraction to a higher plane of expression.
Data Envelopment Analysis (or DEA) is a mathematical model designed specifically for use with nonprofit organizations. Charnes and Cooper report that work on the model began in 1975.(1) Together with Rhodes,(2) they wanted to develop a methodology that would accommodate multiple input/output situations without requiring costing data. Their work resulted in two landmark articles, one providing the mathematical theory and proofs(3) and the other reporting the application of the model to a large-sample experimental project.(4) DEA, like our omniscient salesperson, can simultaneously take into consideration any number of inputs and outputs for any number of decision-making units (or DMUs) and consider all of these factors simultaneously for everyone in the group. The model provides three important pieces of information:
- a series of efficiency scores, which indicate the status of each DMU relative to a designated group of peers;
- identification of any perceived slack (or waste) in the amount of input used or output produced; and
- a set of weights, representing the values attached to each input/output combination by its respective organization, again relative to its peers.
In short, two separate activities occur when DEA is applied. First, DEA weights each input and output, repeating this procedure for each DMU under investigation. Then (should the results indicate that a particular organization is behaving inefficiently), DEA uses these same weights to identify input/output levels that would render that organization's behavior more efficient.
DEA's weighing scheme is at once its key advantage and its most intellectually challenging aspect. Its primary contribution to nonprofit efficiency measurement lies in its abilities to accommodate multiple outputs as well as multiple inputs and to use a series of relative values unique to each DMU. It provides these unique values by basing its comparisons on the results of actual decisions-that is, the input/output values are givens rather than estimates. In other words, instead of using the model to select the values of a specified number of decision variables within certain parameters (i.e., weights), DEA calculates the weights and assigns relative rankings to actual input/output combinations.
The purpose of my current work (and of the dissertation(5) that preceded it) is to measure the technical efficiency with which school media centers transform designated resources into a variety of library-and school-related outcomes. I am particularly interested in determining the ease with which efficiency ratings might be manipulated. My intention is not to characterize library media centers on the basis of their efficiency status-that is, to characterize what makes some more efficient than others-but to look for potential pitfalls in using DEA that could lead to misleading results. My reasons for taking this approach are twofold.
First, like any statistical technique, DEA accommodates whatever data it is given and dutifully assigns each DMU in the sample an efficiency score and an efficiency reference set. It also allows the number and/or types of inputs, outputs, and DMUs to be varied at will. This flexibility, however, is a two-edged sword. While it allows the decision makers in charge of operating the model to target those inputs and outputs they feel are most representative of organizational activity, it also makes it easy for them to control (intentionally or not) the results of the analysis. My own concern is not so much with stamping out the possibility of unscrupulous use-something I, as a researcher, can do little to affect-but with seeking ways of avoiding inadvertent misapplication.
Second, DEA is designed to be used in actual situations in which the decision makers are expected to agree on such things as the number and/or types of inputs, outputs, and DMUs. While there are, of course, certain rules of thumb to be followed, the primary function of DEA recommendations is to serve as points for discussion. My concern here is that, unless one is aware of the conditions under which DEA results may vary significantly, they may be treated as prescribed standards of behavior rather than as avenues for dialogue. In short, I believe it is important to understand how easily the model can be manipulated in order to avoid using it inappropriately.
My dissertation addressed both these points, concluding that results could indeed vary significantly according to how inputs, outputs, and DMUs were grouped and/or included in the model. My current research, consisting of four interrelated projects, continues the original investigation in greater depth. The first two projects look at how changes in variable/DMU groupings affect efficiency status. The third focuses on DEA's unique weighing scheme. The fourth considers how best to incorporate the effects of non-discretionary influences (such as ethnic mix). Taken together, these projects are designed to provide a clearer picture of how DEA works and, in particular, how it might best be used for decision support. While the full results of my explorations to date are too complex to cover here, the following three findings are likely to be of particular interest to an audience of concerned professionals.
First, while there appears to be a relationship between conditional specification (that is, the basis for deciding how participating DMUs are grouped) and efficiency status, the effects of the grouping variable cannot be isolated as easily as one might hope. In other words, it is not entirely clear whether the relationship is intrinsic (i.e., a function of the variable itself), extrinsic (i. e. a natural consequence of the grouping process), or a combination of the two.
Second, there also appears to be a relationship between process specification (that is, the basis for including a variable as either an input or an output) and efficiency status, suggesting in part that how we define efficient behavior may be too easily manipulated by our conceptualization of the problem. In other words, variations in our current understanding of what constitutes an efficient (or productive) school library media center make it difficult for us to engage in comparisons.
Finally, the degree of confidence that can be placed in either of these results is confounded by the variables themselves, which bear little resemblance to what might be called a productive process. In other words, we don't as a role collect data that allow us to assess oar efficiency.
Where, then, does this leave us? We, as library and information professionals, need to take greater advantage of measurement models developed outside the field.
But we also need to become more involved in "boHom-up model enhancement" to adapt these models to our own situations. DEA is clearly a powerful evaluation tool with the potential to summarize complex relationships in a neatly presented comparative fashion. If its results, however, are to validate observed behaviors and not simply the design of the model itself, we must pay closer attention to how these results are to be interpreted. Efficiency experts can at best provide us with recommendations; how best to assess these recommendations is up to us.
Efficiency is both an ideal state and a requisite for survival; in its most general form, it encompasses a range of production possibilities, any one of which can represent a form of efficient behavior. It is my hope that the results of this research stream will help both library media center personnel and school administrators to make efficient choices and, more importantly, to monitor the impact of those choices on the performance of their information centers.
The author gratefully acknowledges the influence of her dissertation advisor, Yale Braunstein.
- A. Chames and W. W. Cooper, "Preface to Topics in Data Envelopment Analysis," in Normalive Analysis in Policy Decisions, Public and Private, ed. Russell G. Thompson and Robert M. Thrall (Basel, Switzerland: J. C. Baltzer Ag, 1985).
- E. Rhodes, "Data Envelopment Analysis and Related Approaches for Measuring the Efficiency of Decision-Making Units with an Application to Program Follow Through in U. S. Education" (Ph.D. diss., Carnegie-Mellon Univ., 1978).
- A. Chames, W. W. Cooper, and E. Rhodes, "Measuring the Efficiency of Decision-Making Units," European Journal of Operational Research 2 (1978): 429-44; 3 (1979): 339.
- A. Chames, W. W. Cooper, and E. Rhodes, "Evaluating Program and Managerial Efficiency: An Application of Data Envelopment Analysis to Program Follow Through," Management Science 27 (1981): 668-97.
- M. S. Easun, "Identifying Inefficiencies in Resource Management: An Application of Data Envelopment Analysis in Selected School Libraries in California" (Ph.D. diss, Univ. of California at Berkeley, 1992).