If you ask a seemingly simple question of law firm or law department lawyers such as, “What was your total cash compensation in the last fiscal year?”, the format of the answers you receive will be far from simple. One person will enter “$100,000,” another “one hundred thousand,” a third will enter “100,000 dollars,” while a fourth will put in “was $100,000.” Although a human knows instantly and correctly how to interpret each variation, they will confound your comparatively rigid computer.
The most common measure in survey results of what statisticians call “central tendency” is the average (the mean) of a set of numbers, such as the average number of paralegals in the responding law firms. The next most common measure is the median, the middle value when the set is sorted from high to low. Rarely you see the mode, which is the most common value in a set of numbers.
My heart is committed to open-source software and code and my abiding faith rests with transparency in research: tell people what you did and how you did it and share as much data as you can so that your investigation is reproducible. Theoretically, others can retrace your steps and recreate your analytic conclusions. Openness and sharing help us all make faster progress in understanding our world.
That credo espoused, however, I must admit that many surveys in the legal industry have sponsors who do not wish to disclose much about their effort.
Here we are considering a different notion than disqualifying a response that you decide is not legitimate. The idea is that a recipient’s answers to a survey – correct and complete though they may be – should not be included in the survey data set. The person competently answered the questions, but they are not a person whose views are sought. For example, ALM’s 2012 survey of alternative fee arrangements explained in the report that 218 law firms responded, of which 194 were qualified.
Here are several clues that suggest a respondent is merely trying to get the report but doesn’t deserve to because their answers were too weak or questionable. Clues can point you to suspects, but it is a good practice to write to them and inquire calmly and objectively about their dubious response. Perhaps they were rushed, misunderstood instructions, or were interrupted while replying. Who knows? Or perhaps they lack genuine bona fides.