Measuring System Performance and Campbell’s Law

by Eric DuBois

This week I would like to discuss measuring system performance and its unintended consequences.  I will share a story out of Larson’s 1987 paper “Perspectives on Queues: Social Justice and the Psychology of Queueing.”

No one particularly likes waiting around.  So naturally one would assume that a good objective measure of the performance of a queueing system is the average time spent waiting in the queue.  Lower the expected time so that people wait less and they will be happier with your service.  Except, of course, that humans are not so easily understood.

In one of my favorite examples of unintended consequences, Larson describes how early morning business passengers at a Houston airport would complain on a regular basis about having to wait for their luggage after deplaning.  The rub being that it wasn’t the wait that irked them so much as it was seeing their fellow bagless passengers proceed straight out the door to the waiting taxis while they waiting on average an additional 7 minutes to receive their baggage.

The baggage handlers were not taking especially long; it was simply that the airport had chosen the most efficient location for the passengers to deplane and leave the airport in the least amount of time.  That is, they chose a location so close to the exit that it only took about a minute from leaving the plane to get to the baggage carousel.  There was simply no way that baggage handlers could keep up.  The solution was simple- increase the distance between the deplaning location and the door.  Now everyone had to waste their time walking from one end of the terminal to the other, but no one felt cheated.  So despite objectively worsening performance, passengers were happier overall and “passenger complaints dropped to nearly zero.”

The point of this tale is a reminder that while our objective functions effect how we view the world and react to it, we shouldn’t forget that there is a very subjective reality aside from any objective measure we use on it.  One of the common worries with EMS service models is that in only providing rewards for serving patients quickly, you will simply abandon some patients to their fate.  Objectively it makes sense- if we only get credit for serving patients within 9 minutes, why bother even serving the patient that is 10 minutes out?  Worse yet, what if all signs point to them not surviving to discharge? should we still rush the nearest ambulance?

Of course we should still serve them.  Our subjective knowledge of customer service and feelings of fairness argue for serving all patients, even if it is not ‘objectively optimal.’

In many cases, the problems that we see stem from Campbell’s Law (Or the closely related Goodhart’s Law), which basically states that if people understand how they are being measured, the measure becomes useless as people seek to do well on the measure.

A common example is school testing.  Back when I taught in Massachusetts, we spent an inordinate amount of time teaching our students how to take the state test (the MCAS).  If they failed the test, they would have a difficult to well-nigh impossible time in graduating high school.  Moreover, we were judged on how many of our students passed their MCAS- a measure which clearly favored the affluent college-bound suburbanites over my urban, vocational students.

I remember on several occasions having to shutdown fruitful discussions on basic scientific principles, because we had to return to the curriculum defined by the state.  The former were clearly more important to their long-term education, but would never be tested on.  So we plodded along and rather than understanding why the Earth has an atmosphere, they learned how to best answer a multiple choice test.  Most of them did pass the MCAS and will be able to graduate high school, but in the true measure- how much they learned in high school, I fear they will be sorely lacking.

So what do you think? Is there better way to measure system performance without skewing the results or losing site of reality?

Advertisements

One thought on “Measuring System Performance and Campbell’s Law

  1. I really enjoyed your story about how the airport relocated the baggage claim to create an illusion of short wait times. It reminds me of how my brother would drive out of his way to avoid waiting at the stoplights in our hometown. While the stoplights did take an unreasonably long time to change, I distinctly remember telling him that it probably didn’t help us get home faster and that it only seemed like it because he was “doing something” rather than just waiting. While that’s just one example, I’d be curious to know if there are other situations where people actively choose a longer overall wait time just so they can feel “busy”.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s